The group said a long-running sanitation crisis in the city of two million meant drinking water was often taken from wells that were contaminated with sewage from broken pipes.
"In many communities there is no water for drinking or bathing, there is sewage in the streets, there is diarrhoea and typhoid and the threat of another cholera epidemic," Tiseke Kasambala, the southern africa director of HRW said.
"Harare's water and sanitation system is broken and the government isn't fixing it."
A 60-page report warned that history showed the risks of doing nothing.
Crumbling infrastructure led to an outbreak of cholera in 2008 which killed over 4,200 people and infected 100,000.
While deadly, the disease is easily preventable with clean water and proper sewage.
Aid organisations have helped to provide medicines and to sink several boreholes in the capital since 2008, but some of the holes have reportedly been contaminated with sewage.
Several suburbs in Harare don't have tap water and people are forced to queue for hours to collect water.
"Harare's water and sanitation system has been destroyed by decades of neglect and by ongoing mismanagement and corruption," Kasambala said.