King's College London psychiatrist David Veale revealed that men who have larger penises are more anxious compared to those with smaller ones.
"Those who have nothing to worry about tend to worry the most," said Veal.
When it comes to women, larger penises are only appealing to women up to a point, said a study published in the Proceedings of the National Academy of Sciences.
The researchers also found women were not more impressed by flaccid penises bigger than three inches in length.
To unravel the penis anxiety, researchers quizzed 173 men at King's College London and online about body image, erectile function and size concerns relating to their penises.
They also managed to find 46 men who agreed to have their penises measured - in both flaccid and erect states - by a urologist.
Veale and his team then measured men's fear of rejection due to penis size, anxieties about being naked around women and other men, and anxiety of penis size being laughed at.
Anxiety seemed higher among older men and gay or bisexual men.
Asked to measure their penises themselves, men tend to exaggerate length.
"This can certainly be attributed to the fragile male ego," said Veale.
However, Veale admits anxieties men carry around is the real problem.
"Quite a few of them have been teased about their size either by an ex-partner or in the showers as an adolescent," he said.
However, women who experienced vaginal orgasms - as opposed to clitoral orgasms - were more likely to be interested in well-endowed guys, said the study.