Under the new legislation, the entry for gender can be left blank on birth certificates, effectively creating a category for indeterminate sex in the public register.
But activists promoting the rights of so-called 'intersex' people said they hoped the creation of a third gender option would open the door to broader changes limiting genital surgery on newborns with both male and female characteristics.
"It's a first, important step in the right direction," Lucie Veith, an intersex person from the northern German city of Hamburg, told AFP.
But Veith said leaving the gender undefined on birth certificates was never the main lobbying point for her group, the German chapter of the Association of Intersexed People, or others in the intersex community.
"That we forbid cosmetic genital surgeries for newborns, that is our first demand," Veith said.
The German law is intended to remove pressure on parents to quickly make a decision about controversial sex assignment surgeries for newborns, but many advocates say it does not go far enough.
"The surgeries are likely to continue in Germany," said Silvan Agius, policy director at ILGA Europe, a lesbian, gay, bisexual, trans and intersex rights group.
"Parents can already refuse these surgeries," Agius added.
"You can already say, 'No, thank you very much, I don't want any surgery until my child can choose his or her gender.'"
Operations on intersex babies and infants in many European countries take place without adequate informed consent by the patients, according to a 2012 European Commission report on the topic.
The report also found that many adults born intersex are angry these surgeries were performed without their consent.
Experts estimate one in 1,500 to 2,000 births result in a baby of indeterminate gender or both male and female gender features.
The Council of Europe for the first time last month addressed the issue, in a Parliamentary Assembly resolution calling on member states to study the prevalence of "non-medically justified operations" that may harm children and take steps to "ensure that no-one is subjected to unnecessary medical or surgical treatment that is cosmetic rather than vital for health during infancy or childhood".
Agius and other advocates worry that the new German law creates a de facto third gender category legally, but does nothing to change a society that operates largely on a gender binary with facilities such as separate male and female public toilets.
"There could be many other laws that could follow it and make it implementable and good," Agius said.
"My point is that if it remains as is... then it's greatly deficient."