The two men and two women who met with Pope Benedict XVI were nominated by the church's Professional Standards Office, a body established to deal with responses to church-related abuse complaints.
Anthony Foster, whose two daughters were raped by a priest, and who travelled to Sydney from the UK in the hope of an audience, said it was "a slap in the face" not to have been included in the meeting.
"Somehow they've managed to handpick ... victims and haven't even had the courtesy to speak to us," he said. "We wanted to have these discussions so we can move the church forward and instead they've treated us with the utmost discourtesy."
He said he was told late last night that there would be no meeting with victims.
Victims support group Broken Rites said it was "not a good look" for the Fosters to be excluded. "The meeting was purely public relations. It's all about just managing the crisis," spokesman Bernard Barrett said on Fairfax radio.
Meanwhile, Pope Benedict XVI consoled four victims of sex abuse by the clergy on Monday morning, after celebrating a private mass for them in Sydney.
The victims and their supporters joined the Pope for mass at St Mary's Cathedral before meeting with him for up to half an hour in a last minute addition to the 81-year pontiff's itinerary.
The Pope met the two men and two women to hear about their ordeals and console them, papal spokesman Father Frederico Lombardi said.
"With regard to abuses by members of the clergy, His Holiness Pope Benedict XVI on Monday morning celebrated mass with a group of representatives of victims of sexual abuse," Lombardi said.
"The Pope listened to their stories and consoled them. He assured them of his spiritual proximity and proposed to continue to pray for them, for their families and for all victims," news.com.au quoted him, as saying.
Lombardi said Benedict spoke individually with each victim in an atmosphere of "respect, spirituality and intense emotions."
The Archdiocese of Sydney, headed by Cardinal George Pell, said the meeting showed the church's commitment to bring healing and justice to those who have been so terribly hurt by sexual abuse.
On Saturday, the Pope issued an historic apology for what he described as the "evil" of priestly sex abuse, saying he was "deeply sorry" and calling for those responsible to be punished.
The Pontiff is flying back to Rome on a Qantas charter plane that left Sydney Airport at about 10.30a.m. (AEST).