The proposal for religious same-sex marriage was rejected by the protestant Church of Norway, even though it had the support of most of the country's bishops.
Norway was among the first countries in Europe to grant homosexuals full rights, including marriage and adoption in 2009, but the Church does not marry same-sex couples.
Eight of Norway's 12 bishops said in October they favoured such a move, but on Tuesday the Church's highest decision-making body the synod rejected the proposal.
"We are disappointed," said Baard Nylund from a national lesbian, gay, bisexual and transgender rights group.
"It's sad that the Church did not seize the occasion to be the Church of the people -- open and generous -- that it says it wants to be," he told AFP.
Delegates at the national synod also rejected proposals to allow priests to bless a gay marriage on the sidelines of a civil ceremony.
But they also voted against a proposal to maintain the status quo and reserve marriage for heterosexual couples, plunging the synod into chaos.
"It (the rejection of all options) is something that no one had foreseen and no one knows now what will happen," bishop Tor Berger Joergensen told public broadcaster NRK.
"We must have a little time now to look into the procedures."
In neighbouring Sweden, the Lutheran church approved same-sex marriages in 2009.