The drug makers said Friday that 24 of 741 volunteers who got the vaccine in one segment of the trial later become infected with HIV, which causes AIDS.
In a comparison group of volunteers who got dummy shots, 21 of 762 participants also became infected with HIV.
"It's very disappointing news," said Keith Gottesdiener, head of Merck's clinical infectious disease and vaccine research group. "A major effort to develop a vaccine for HIV really did not deliver on the promise."
The study volunteers were all free of HIV at the start of the experiment. But they were at high risk for getting HIV: most were homosexual men or female sex workers. They were all repeatedly counselled about how to reduce their risk of HIV infections, including use of condoms, according to Merck.
In a statement, the US National Institute of Health (NIH), which partly funded the study, said a data safety monitoring board, reviewing interim results, found the vaccine could not be shown to prevent HIV infection or limit severity of the disease "in those who become infected with HIV as a result of their own behaviours that exposed them to the virus."
The Merck vaccine was the first major test of a new strategy to prevent HIV infection.
The first wave of attempts to develop a vaccine tried to stimulate antibodies against the virus, but that didn't work.
The new effort — and one being tried in most other current research — is aimed at making the body produce more of a crucial immune cell called killer T-cells. The goal is to simultaneously "train" those cells, like an army, to quickly recognize and destroy the AIDS virus when it enters cells in the bloodstream.
Merck and the HIV Vaccine Trials Network, an international collaboration of researchers and institutions, co-sponsored the study. The Merck vaccine was the farthest along of several in development, and this was the first large-scale test of it.
The experiment, called STEP, began in December 2004 and had enrolled 3,000 volunteers in Australia, Brazil, Canada, the Dominican Republic, Haiti, Jamaica, Peru, Puerto Rico and the United States.
It aimed to determine if the vaccine, a shot made from a weakened cold virus that included bits of three HIV genes, could prevent HIV infection, or reduce the amount of HIV in the blood of infected people, or both.