The seven pensioners include a 96-year-old and her friend who dedicated 40 years of her life to the NHS.
Housing Association bosses have now warned the group the four benches they sit on in Mottingham will be removed if they don't tone down the volume of their conversations.
The pensioners wail they don't do anything more than talk about such everyday things like the weather, their families and the cost of living.
Ann Reddy, 69, who is recovering from a stroke and has had 45 operations, said she was horrified about being branded a trouble-maker.
The retired medical secretary, who worked for the NHS for 40 years, labelled the accusations "unbelievable."
The pensioner, who suffers from rheumatoid arthritis, said: "How could I possibly be capable of anti-social behaviour?
"When I told my doctor that we might be having our benches taken away, he asked me if I had been drinking.
"We don't drink and sit on walls throwing cans of lager around the place. We don't sing in the middle of the night. It's unbelievable."
She added: "We just love sitting outside in the fresh air enjoying each other's company and talking quietly about our families, the weather, and the cost of living.
"But someone has complained and said we are making too much noise."
Her friend, 96-year-old Rose Anderson, a great-grandmother of nine great, said she would "go mad" if forced to sit in her flat all day by herself.
She said: "I'm so old now and I have got nothing else to do with my day.
"We all deserve to be treated with a lot more respect than this."
But Broomleigh Housing Association are refusing to back down.
Julie Schoon, assistant director of supported housing at Broomleigh, said: "As a registered social landlord we are responsible for ensuring that any complaints of noise nuisance or other forms of anti-social behaviour are acted upon.
"Following a number of complaints from residents over the last two years we have worked hard to try and mediate between those involved.
"We are very reluctant to remove any of the benches and would view this as a last resort. We are currently talking to residents and considering various solutions to try and resolve the issue."
Amy Swan of Help the Aged said: "It's such a tragic state of affairs when older people tell us that the only person they see from week to week is the postman."
A Help The Aged report last week found that a third of people over 65 in the UK - 3.6 million in total - now live alone and have little contact with their friends and family.
Latest figures show more than a million pensioners half a million leave their house less than once a week, and that 300,000 feel like prisoners in their own homes.