The technique can more confidently detect the genetic signatures of interbreeding than previous approaches and will be useful for evolutionary studies of other ancient or rare DNA samples.
Study co-author Konrad Lohse, a population geneticist at the University of Edinburgh, said that their approach can distinguish between two subtly different scenarios that could explain the genetic similarities shared by Neandertals and modern humans from Europe and Asia.
The first scenario is that Neandertals occasionally interbred with modern humans after they migrated out of Africa. The alternative scenario is that the humans who left Africa evolved from the same ancestral subpopulation that had previously given rise to the Neandertals.
Many researchers argue the interbreeding scenario is more likely, because it fits the genetic patterns seen in studies that compared genomes from many modern humans. But the new approach completely rules out the alternative scenario without requiring all the extra data, by using only the information from one genome each of several types: Neandertal, European/Asian, African and chimpanzee.
The authors originally developed the method while studying the history of insect populations in Europe and island species of pigs in South East Asia, some of which are extremely rare.
The study has been published in the journal GENETICS.