Watching the small screen for prolonged periods is also bad for your heart, according to the research published in the Journal of the American Heart Association.
"Compared to people who watch less than two hours of television per day, people who watch more than four hours per day have a 46 percent higher risk of death from all causes," researcher David Dunstan told AFP.
They also have an 80 percent increased risk from cardiovascular disease, he said.
Sitting down for long periods stops the body from using its muscles and adequately processing sugars and fats, Dunstan said.
The findings come from a six-year study into the viewing habits of some 8,800 Australians which stripped out the influence of other health factors such as age, sex, smoking, weight and exercise.
Australians watch an average of three hours a day, said Dunstan, a researcher with the Melbourne-based Baker IDI Heart and Diabetes Institute.
The study found that even people who exercised regularly were potentially hurting their bodies by sitting down for extended periods, he said.
"What these findings suggest is that in addition to regular exercise, people need to consider avoiding prolonged periods of sitting; whether it's in front of the television, working long hours or driving for prolonged periods.
"And for TV, a specific message could be: switch off, stand up and keep moving," the researcher said.