A commissioner should be appointed to protect and promote the rights of people with learning disabilities in England, said a report.
Sir Stephen Bubb, the chief executive of the Association of Chief Executives of Voluntary Organisations, who has been reviewing the sector since the abuse scandal at Winterbourne View home, also said care homes "have to close".
In 2011, the BBC's undercover documentary team filmed abuse of patients with learning disabilities at the Winterbourne View private hospital near Bristol.
In a report published by Sir Stephen in November 2014, 'Time for Change' he said that many people were being kept in hospitals far from home for far too long.
He made 10 recommendations, including closing large "inappropriate in-patient facilities" for care services for people in their own community.
He has demanded the Government to appoint a learning disabilities commissioner for a charter of rights to be drawn up for people with learning disabilities and their families.
Sir Stephen Bubb said, "I am calling for an office of a learning disabilities commissioner to be established. Just as a children's commissioner was established following the Victoria Climbie Inquiry, there is a firm argument for establishing this post. It would have a statutory duty to promote and protect the rights of all people with learning disabilities and their families."
"I have spoken directly to people whose experience of these services goes back far beyond 2011 and Winterbourne View. So this report expresses the views and experiences of the people most affected by change."
"I am still shocked by the way we as a society have condoned the poor or abusive treatment of some of the most vulnerable people in our society."
He said services in England still have "a long way to go before the system can be trusted, and we still have a long way to go in convincing people with learning disabilities that change will happen".
He warned that at least 1,300 people will move out of hospital care by 2019. There is a critical need to deliver housing for the vulnerable. An extra 10,000 staff will be needed to support people with learning disabilities in their own community.
The report stated that housing for people with learning disabilities should be exempt from proposed housing benefits caps.
Jan Tregelles, chief executive of learning disability charity Mencap, and Vivien Cooper, chief executive of the Challenging Behaviour Foundation, said, "now is the time to restore people's faith in the system."
"People with a learning disability and their families have endured nearly five years of failure by national and local government, and the NHS to bring about meaningful change for the 3,500 people in in-patient units.This has left people often far from home in units where they are at increased risk of abuse and neglect, their families fighting to bring loved ones home."
"This is taking place as the NHS and local government are spending an estimated Ģ600 million a year on buying the wrong kind of care for thousands of people. Now is an opportunity to restore faith, where so far there has been repeated failure, and build the community support people with a learning disability and their families want and have been promised."