The vaccine trial, known as RV144, involved more than 16,000 HIV negative volunteers in Thailand and used two investigational vaccines in combination.
When its results were first published in 2009, experts hailed its 31 percent protection rate as a pioneering achievement, even though it fell short of hopes by providing only a partial shield against HIV.
A vaccine would have to offer 50 percent protection in order to be offered to the public.
Experts are continuing to study the data for clues on what went wrong, and now they believe that an antibody that was induced by the vaccine, immunoglobulin A, was fighting against another that would have blocked the virus.
"We learned that a specific vaccine-induced immunoglobulin A can weaken the protective effect of immunoglobulin G," said Georgia Tomaras, director of the Laboratory of Immune Responses and Virology at Duke Human Vaccine Institute.
"IgA competes with IgG to bind to the same site on the virus's outer envelope that is exposed on infected cells," said Tomaras, lead author of a study in the Proceedings of the National Academy of Sciences that describes the research.
"We found that the IgA antibodies can block the activity of natural killer cells activated by IgG, further interfering with the vaccine-induced immune response."
Tomaras said the latest findings offer clues toward how to induce more effective responses in future vaccines.
There is currently no vaccine against HIV on the market, and no cure for AIDS, which has killed some 35 million people around the globe.
According to the World Health Organization, 34 million people worldwide were living with HIV at the end of 2011.