The case is being closely watched by Malaysia's ethnic Chinese and Indian minorities who say their rights are being eroded by rising "Islamisation" in a country where 60 percent of the population is Muslim Malay.
Secret conversions of children, which can deprive the non-Muslim parent of custody, and "body-snatching" cases where Islamic authorities tussle with families over the remains of people whose religion is disputed, have raised racial tensions.
The Federal Court was asked to hear the case of Hindu woman S. Shamala who fled to Australia in 2004 with her two young sons after her husband Muhammad Ridzwan Mogarajah converted to Islam and secretly converted the children.
The High Court in 2004 gave Shamala custody of the children on condition she raised them as Muslims, an order Mogarajah appealed in the civil courts, and in the religious courts which operate in a dual-track system in Malaysia.
Under sharia law, a non-Muslim parent cannot share custody of converted children. Non-Muslims also complain that they do not get a fair hearing when such cases end up in the religious courts.
"The significance is as to whether or not in Malaysia under the federal constitution, one parent can convert the religion of the child of the marriage to another religion without the consent of the other parent," Shamala's lawyer Cyrus Das said.
The Federal court delayed hearing the case, saying that it first had to rule on a preliminary objection by Mogarajah's lawyers that it should not proceed unless the children were returned to Malaysia.
Rights groups criticised the decision to delay, saying that overlaps in jurisdictions between the two legal systems need to be addressed urgently.
"There is a huge crack in the nation due to the overlaps and the rolling back of rights. (If unresolved), people will have no choice but to go to the ballot box," Ivy Josiah from the Women?s Aid Organisation said at the court.
The government announced last year that legislation would be amended so that children's conversion required the consent of both parents, but the reform has been stalled pending consultations with the Malay royal rulers.