Today, Wu Pao-chun is a world-famous baker and a driving force in a culinary revolution sweeping Taiwan although as a boy often had to eat leftovers.
Wu, 42, won the bread section in the prestigious Bakery Masters in Paris last year, to much fanfare back home, and now his loaves sell like hot cakes in a society more accustomed to steamed buns.
"I have very strong attachment to baking. To me, bread is art, much more than just food," Wu said in an interview with AFP.
It is an art with a growing number of connoisseurs on this island of 23 million, judging from sales at his bakery in south Taiwan's Kaohsiung city.
Every day he and his assistants bake 300 loaves of his trademark bread filled with the popular fruit lychee, and a similar amount of bread with longan fruit. They are snatched up before they grow cold.
Wu, who has written two books, is featured in several TV programmes shown on lifestyle channels across the world and customers from as far away as United States ask to have their pictures taken with him.
It is a success that has not come easily to Wu, whose father's death in a traffic accident forced his mother to bring up eight children in abject conditions.
"Very often, when she was hungry, she would get vegetables or leftovers from family and neighbours," Wu said, the memory bringing tears to his eyes.
At the age of 15, Wu went to Taipei to start an internship at a bakery, but found life hard and unrewarding, as his school had left him a functional illiterate with not even basic calculating skills.
The turning point came when he joined the army as a conscript and a friend taught him how to read. This kindled a thirst for knowledge, and he travelled to Japan to hone his baking skills with masters in the field.
Japan ran Taiwan as a colony for 50 years after 1895 and introduced the island to bread, albeit in a soft version without hard crusts.
"For decades, people in Taiwan preferred soft bread rather than the European-style hard-crusted bread. But the taste has gradually changed," he said.
Hard-crusted bread now accounts for about 30 percent of the bread consumers buy in Taiwan -- a transformation that Wu has done his bit to help bring about.
The path from illiterate farm boy to world-class baker was not an obvious one given his background -- most of Wu's siblings are factory workers or housekeepers -- and he knows exactly who to thank for his success.
"The most important influence bringing me where I am today is my mother. She never gave up and never bowed to pressure, no matter how poor our family was," he said.
Wu's bond with his mother is a captivating story in a society where respect for the elderly may be dwindling, and famed local movie director Lin Cheng-sheng plans to shoot a film about the baker and his mother next year.
Despite having made it further than most in his profession, Wu sees no reason for complacency. And having no obvious rivals does not relieve him of the pressure to perform.
"My real competitor is myself. I know if I want to keep growing, I have to force myself to think in new ways all the time," he said.