The sex selection technique produces a generation of mosquitoes which is 95-percent male, as opposed to 50 percent in normal populations, they reported in the journal Nature Communications.
So few females are left that the mosquito population eventually collapses, curbing the risk to humans from the malaria parasite that the blood-feeding females transmit.
"Malaria is debilitating and often fatal and we need to find new ways of tackling it," said study leader Andrea Crisanti, a professor at Imperial College London.
"We think our innovative approach is a huge step forward. For the very first time, we have been able to inhibit the production of female offspring in the laboratory, and this provides a new means to eliminate the disease."
Malaria kills more than 600,000 people each year, with young children in sub-Saharan Africa on the frontline, according to the UN's World Health Organisation (WHO).
The result of six years' work, the method focuses on Anopheles gambiae mosquitoes, the most dangerous transmitters of the malaria parasite.
- 'Super cool work' -
The scientists injected a stretch of enzyme DNA into the genetic code of male mosquito embryos. The modification essentially shreds the X chromosome during sperm production in adulthood.
As a result, almost no functioning sperm carried the X chromosome, which determines female offspring. Instead, most sperm carried the Y chromosome, which produces males.
The modified mosquitoes were put to the test in five cages, each containing 50 genetically modified males and 50 normal, wild females.
In four of the five cages, the entire population was wiped out within six generations because of the progressively greater shortage of females.
Modified male mosquitoes produced only modified male heirs, which did the same until there were no females left.
"The research is still in its early days, but I am really hopeful that this new approach could ultimately lead to a cheap and effective way to eliminate malaria from entire regions," said Crisanti's colleague, Roberto Galizi.
In an independent comment, University of Oxford specialist Michael Bonsall said the research was "super cool work."
"This has important implications for limiting the spread of malaria," he told Britain's Science Media Centre. "It will be very exciting to see how this specific technology is now taken forward."
Scientists are already experimenting in the wild with Aedes aegypti mosquitoes -- which carry dengue fever -- that have been modified to create offspring that do not reach adulthood.
They survive for just a week, compared to a month for normal mosquitoes.
Brazil and Malaysia have already released batches of these insects, and Panama in January said it would follow suit.
The programmes have run into concerns from environmentalists, who point to unknown impacts of GM releases on biodiversity balance.
If one mosquito species is eliminated in a neighbourhood, this opens up opportunities for a rival, and potentially dangerous, species to move in, they say.