For the study, more than 5,000 middle-aged men were interviewed about their drinking habits three times over 10 years.
Then, they underwent memory and other cognitive tests beginning at an average age of 56. These tests were repeated twice over the next 10 years.
"Our study focused on middle-aged participants and suggests that heavy drinking is associated with faster decline in all areas of cognitive function in men," said study author Severine Sabia of the University College London.
The mental abilities of heavy drinkers declined between one-and-a-half to six years faster than those who had fewer drinks per day.
Men who drank 36 grams of alcohol or more per day saw the steepest declines in their memory and brain function.
Some 2,000 women were also included in the study, but there were not enough heavy drinkers among them to analyze their rates of mental decline compared to moderate or non-drinkers.
Neurology is the medical journal of the American Academy of Neurology.