The ruling applies to 179 parks across the state, as well as beaches, sporting grounds, picnic spots, camping areas and historic sites run by the Office of Parks, Recreation and Historic Preservation.
A New York-based group, Citizens Lobbying Against Smoker Harrassment (C.L.A.S.H), had challenged the ban, on the grounds that parks had exceeded their authority.
New York State Supreme Court Justice George Ceresia ruled in their favor, declaring that the state parks agency had "extended their reach beyond interstitial rulemaking and into the realm of legislating."
C.L.A.S.H on Thursday welcomed what it called a vindication of individual freedoms.
"This is putting the prohibitionists on notice that, despite their ugly war being waged on adults who choose to smoke, they are not entitled to a free-for-all in governing when it comes to this segment of society," said its founder Audrey Silk.
It has been illegal to smoke in bars, restaurants and work places in New York state for the last 10 years.
In New York City it is also illegal to smoke on public beaches, parks and public swimming pools since 2011.
That separate ban was not included in this lawsuit.