The Philharmonic, famed for its annual New Year's concert, sent a letter to the descendants of the painting's former French owner Marcel Koch on Thursday to inform them of the move, a spokeswoman said.
The 1883 painting "Port-en-Bessin" by French artist Paul Signac, known for his pointillistic style, was confiscated in 1940 in France's Jura region and given to the Philharmonic after it performed in the region, the orchestra said in a statement.
After over two decades of research, an art historian commissioned by the orchestra, Sophie Lillie, finally managed to identify the painting's origins.
"We're very happy that thanks to Sophie Lillie's efforts a breakthrough was finally achieved and we found the descendants," Philharmonic musician and president Clemens Hellsberg said in the statement.
"The restitution of this painting was of major concern to us. We've tried for years to deal with the Vienna Philharmonic's past and we want to face our responsibility in making up for past injustice."
A ceremony will be held to return the painting to the family of Marcel Koch, but it was still unclear when this might occur.
The Philharmonic meanwhile said it would continue to search its archive for further stolen artifacts.
Austria, which long considered itself a victim of Nazi Germany and only recently admitted shared responsibility in World War II and the Holocaust, has regularly been at the centre of the debate over the restitution of looted Nazi art.
In its most high-profile case in 2006, the state had to return Gustav Klimt's prized "Portrait of Adele Bloch-Bauer" to the family of the former owners after a years-long tug-of-war.
Last year, a Jewish family also laid claim to one of Austria's most famous artworks -- Klimt's "Beethoven Frieze" -- arguing the state had forced its sale after the war.