Now the visitors are coming back, but this time they want to see the devastation and the monuments to those who died, the latest example of a phenomenon dubbed "dark tourism" where holidaymakers pay to witness the aftermath of others' misery.
"You can't really get a sense how huge the tsunami was unless you actually come here and see," said Akira Shindo, 15, from New York, on a recent tour of part of Japan's devastated northeast coast.
More than 18,000 people were killed when a 9.0-magnitude undersea quake sent huge waves barrelling into Japan.
Whole communities were destroyed, buildings turned into matchwood and acres of prime land left unfarmable when a furious sea smashed ashore.
In Rikuzentakata, a forest of 70,000 pine trees that had protected the city from ocean winds for 300 years was swept away.
Just one tree -- the "miracle pine" -- survived the ravages of nature. It has undergone 150 million yen ($1.5 million) of reinforcement to prop it up and has now become a must-see spot for visitors to the area.
"The tree was the tallest, 27 metres- (89 feet-) high, and the two storey-building behind it prevented the wave from sweeping it away," said Mitsuko Morinaga, a 62-year-old volunteer who takes tourists around her devastated home town.
Near the pine tree, dozens of excavators and dump trucks work busily, processing piles of debris or clearing land for new homes.
"Reconstruction is under way, but evacuees have to wait at least three years before their new houses will be built on the hillside," Morinaga said.
Travel agent Shuichi Matsuda, who organised the tour for 24 people, said he set it up because he "wanted to prevent the memory of the disaster from fading".
Everyone AFP spoke to on the tour of Rikuzentakata expressed horror at the suffering of people whose lives were torn apart by the tragedy.
But disaster zones undoubtedly draw their fair share of ghoulish sightseers.
Seven years after Hurricane Katrina devastated the United States city of New Orleans, residents of one hard-hit area became so fed up with the hordes of gawking tourists feasting on their misery that they got the city to ban tour buses.
A hand-painted sign on one street corner summed up their frustration in dripping black paint: "Tourist -- shame on you. Driving by without stopping. Paying to see my pain. 1,600 died here."
Lauren Cason, a spokeswoman for the city's tourism board said visitors were welcome to New Orleans, but residents wanted them to see the positive side.
"What we try to highlight is the comeback story and that the city is now thriving," she told AFP.
While the bigger buses are gone, the voyeurs are not.
Two of the city's more than 30 tour operators still offer limited "Katrina" tours and a fair number of the Big Easy's nine million annual visitors also opt to take a taxi or rent a car to see the remaining wreckage.
-- Respectful tourism please --
Residents in Christchurch, New Zealand, where 185 people died in a February 2011 earthquake that flattened the downtown area, have grown accustomed to buses disgorging camera-wielding visitors at sites such as the ruins of the Anglican cathedral, once the symbol of the city.
Shelagh Ferguson and Alex Coats, marketing researchers at New Zealand's Otago University, last month published a study on the dark tourism phenomenon which found locals accepted such interest was inevitable, but wanted strict controls to prevent stirring more trauma in a community where memories of the disaster remain raw.
The locals resented "rubbernecking" in suburbs where people are still rebuilding their homes two-and-a-half years on. But they had no problem with tours taking in sites in Christchurch's central business district where mass fatalities occurred, provided they were respectful and avoided sensationalism.
The study, carried out using in-depth focus groups, said as memories of the quake faded outside Christchurch, disaster tours served to remind incoming visitors about its victims and what the city had endured.
"We found that residents understood the fascination that death and disaster might exert over visitors and should not be ignored as confrontation with death allows for catharsis, acceptance and a means of grieving" it concluded.
On a very practical level, tourists spend money -- often at a time when a devastated area desperately needs jobs and investment to get back on its feet.
Akira Oikawa, who sells fish, seaweed and other processed marine products, said the post-disaster day-trippers to Rikuzentakata and the nearby area were helping to make up the shortfall.
"We are grateful for tourists visiting here and buying local products, as we saw a drop in the number of tourists after the disaster," he said.
"But it's hurtful when people ask casually about how many people died," he added. "We appreciate a little bit of empathy."