The study, by researchers at the University of Liverpool, says there is a genetic basis for avoiding inbreeding so as to prevent what is called "inbreeding depression", or the emergence of the faulty traits.
The findings of the study have been published in the online edition of the journal Current Biology.
In experiments with house mice, the researchers found they relied on a set of specially evolved proteins in their urine -- major urinary proteins (MUPs) -- to identify relatives and avoid mating with them.
"Mice use these variable proteins as a kind of genetic barcode that normally differs between individuals," said Jane Hurst, who led the study.
"Animals with the same sets of proteins can recognise each other as relatives (through their scent), and so avoid mating with each other.
"By simply checking the match between their own urine proteins and those of any animal they meet, they will be able to identify many of their closest relatives, even if they have never met them before."