During the study, the researchers genetically engineered mice to carry a mutation in a gene called Pet-1, which directs the development of the brain serotonin neurotransmitter system.
Though Pet-1 deficient mice mothers showed normal rates of pregnancy and gave birth to normal numbers of offspring, it was observed that not of their offspring survived to five days of age.
Further studies indicated that a specific deficiency of maternal care was the cause of pup mortality.
Essential maternal care includes construction of proper nests and huddling of offspring in the nests for warmth.
The researchers found that Pet-1 deficient mothers, though nursed their offspring, often failed to build suitable nests and never organized offspring in a huddle.
They said that offspring neglect led to death from cold exposure.
The study also showed that a partial restoration of Pet-1 function in the developing brain of females partially restored their serotonin levels, and maternal behaviour in adulthood.
The finding indicated that subtle changes in the embryonic formation of the brain serotonin system in females could impact the quality of the maternal care they later provide for their offspring.
The researchers say that future studies with Pet-1 deficient mothers may help to further elucidate the link between serotonin and maternal behaviour, and lead to the development of new therapeutic approaches for treatment of post-partum depression and child neglect.
The study has been reported in the journal Nature Neuroscience.