Lower brain volumes have been linked to progression of dementia and problems with thinking, learning and memory.
In the study involving 1,839 adults, the participants underwent magnetic resonance imaging (MRI) and a health examination.
"Most participants reported low alcohol consumption, and men were more likely than women to be moderate or heavy drinkers," the authors write.
"There was a significant negative linear relationship between alcohol consumption and total cerebral brain volume," they added.
The research team led by Carol Ann Paul, M.S., of Wellesley College also found that although men were more likely to drink alcohol, the association between drinking and brain volume was stronger in women.
This could be due to biological factors, including women's smaller size and greater susceptibility to alcohol's effects.
"The public health effect of this study gives a clear message about the possible dangers of drinking alcohol," the authors write.
"Prospective longitudinal studies are needed to confirm these results as well as to determine whether there are any functional consequences associated with increasing alcohol consumption.
"This study suggests that, unlike the associations with cardiovascular disease, alcohol consumption does not have any protective effect on brain volume," they added.
The report appears in the October issue of Archives of Neurology, one of the JAMA/Archives journals.