In the UK the Department of Health recommends that pregnant women and those trying to conceive should not drink alcohol at all and no more than 1-2 units a week.
Middle class women were most likely to drink more than this, the study found.
The researchers base their findings on responses to food frequency questionnaires by 1264 women at low risk of birth complications in Leeds. All the women were part of the Caffeine and Reproductive Health (CARE) study, looking into links between diet and birth outcomes.
The mums to be were asked how often they drank alcohol, and what type it was, at four time points: in the four weeks before conception; and in each of the subsequent three months (trimesters) throughout the pregnancy.
Alcohol consumption was significantly higher before conception and in the first three months of pregnancy than in the subsequent two trimesters, averaging 11, 4, and just under two units a week, respectively.
Over half (53%) of the women drank more than the recommended maximum two weekly units during the first trimester. And almost four out of 10 said they drank more than 10 units a week in the period leading up to conception.
Those who drank more than two units a week were more likely to be older, educated to degree level, of white ethnicity, and more likely to live in affluent areas.
Some 13% of the babies born were underweight, and 4.4% were smaller than would be expected; a similar proportion (4.3%) were born prematurely.
Drinking during the first three months of pregnancy was most strongly linked to these outcomes.
Women who drank more than the recommended two weekly units were twice as likely to give birth to an unexpectedly small or premature baby than women who abstained completely.
But even women who didn't exceed the maximum recommended alcohol intake during this period were still at increased risk of a premature birth, even after taking account of other influential factors.
Drinking during the period leading up to conception was also linked to a higher risk of restricted fetal growth, indicating that this may also be a critical period, suggest the authors.
"Our results highlight the need for endorsing the abstinence-only message, and further illuminate how timing of exposure is important in the association of alcohol with birth outcomes, with the first trimester being the most vulnerable period," they write.