The comedy legend, who is in the early stage of Alzheimer's, can only meet fans at Abbotswood nursing home in Ballasalla after they take his family's permission.
His family - son Nicholas, 55, daughter Jacqueline, 52, and daughter-in-law Kim, 53 - insisted they had his best interests at heart while keeping him away from his Isle of Man mansion which houses his beloved showbiz memorabilia.
The 92-year-old's pals Malcolm and Enid Watson, who have known the star for 27 years, however complained that the actor has been confined to a spartan room with plastic furniture and claimed that they have been frozen out of his life.
"I was told in no uncertain terms to keep my nose out of things. We can't understand why he's been shuffled off to a tiny room from a home he loved so much," the Mirror quoted Watson, as saying.
"Some days Norman is perfect. At other times he's very forgetful. But there's no reason why he can't have 24-hour care in his own home. That's where he wants to be," Malcolm said.
The actor's daughter-in-law Kim disputed Watsons' description of conditions at the home and claimed that the family had taken his favourite chairs and paintings to the nursing home.
The star's former housekeeper Maureen Farragher scared that he will be forgotten by people.
"I'm so frightened he might be forgotten," Farragher said. Kim also hit out at the criticism the family had to face for taking care of the actor.
"No family can have cared more than us. Norman has vascular dementia and is getting confused. Our sole motivation is his well-being," she said.
"A lot of people in his life have their own agenda. They want to see him and no explanation of why they can't is going to change their mind," she added.
Wisdom's longtime agent Johnny Mans also insisted that the family was not to be blamed and the actor was happy.
"He's jolly and happy. The children want to give him the chance to settle in at the home gently," Mans said.
The actor's family had earlier admitted selling off some of his luxury cars while he was away on a cruise saying that a doctor advised that Wisdom should not drive anymore.
Wisdom's home, Ballalough near Ramsey, is also expected to be sold off to fetch money for his care.