Parents or caregivers were asked about milk consumption when the infant was two and were questioned again two years later, when the child was again weighed and measured.
Overweight or obesity was widespread: 30.1 percent of the children at two years fell into this category, rising to 32.2 percent at the age of four.
But children who were overweight or obese were likelier to drink skimmed milk or semi-skimmed milk, which has one-percent butterfat, than counterparts of normal weight, the probe found.
Low-fat or fat-free milk was consumed by 14 percent of heavy two-year-olds and 16 percent of heavy four-year-olds.
This compared with nine percent of normal-weight two-year-olds and 13 percent of normal-weight four-year-olds.
Kids who drank full-fat milk, which has a 3.5-percent fat component, or reduced-fat milk, which has two-percent fat, also tended to weigh less than counterparts who drank skimmed or semi-skimmed.
US health watchdogs -- the American Academy of Pediatrics and the American Heart Association -- recommend that all children drink skimmed or semi-skimmed milk after the age of two to reduce intake of saturated fat.
The study says that the logic behind this recommendation is to reduce consumption of calories and thus prevent weight gain.
But the reality could be more complex, it cautions.
Milk fat may increase a sense of fullness, thus reducing craving for fatty or calorie-rich foods, the authors argue.
Obesity fighters, they argue, should look at other sources of weight control, "such as decreased television viewing, increased physical activity and decreased juice and sugar-sweetened beverage intake, as well as a focus on non-Western diets with higher vegetable content."
The paper, published on Monday in the journal Archives of Disease in Childhood, was led by Mark DeBoer at the University of Virginia in Charlottesville.