So-called low-grade gliomas grow slowly in the brain and may be discovered only when seizures, personality changes or weakness become apparent.
They strike approximately one in 125,000 adults and one in 170,000 children.
Treatment options include surgery, chemotherapy and radiotherapy, and the new findings may help doctors strike a better balance between the benefits of radiation treatment and its undesirable side-effects.
Earlier studies had shown that reduced mental abilities in patients who had survived low grade gliomas for six years was due mainly to the brain cancer itself and not to the treatment.
Beyond that, the possible long-term impact of the therapy was still unknown.
In a study lead by Linda Douw at VU University Medical Centre in Amsterdam, researchers compared the mental functions of two groups of people who had each remained stable, on average, 12 years after being diagnosed with brain cancer.
One group of 32 patients had received radiotherapy, and the other, with 33 patients, had been given other forms of treatment, notably chemotherapy.
The tests covered attention span, verbal and working memory, the speed with which information was processed, physical coordination and "executive functioning" related to planning and decision-making.
Just over one quarter of the non-radiation patients showed measurable cognitive disability compared to more than half -- 53 percent -- of those who had undergone radiotherapy.
"Radiotherapy is associated with long-term cognitive deterioration, regardless of the dose," the authors write in the British medical journal, The Lancet.
"All surviving patients who had radiotherapy are at risk of developing attentional problems."
The researchers say their findings suggest that delayed, rather than early, treatment may be a better option, especially for patients who have undergone surgery.
Other scientists familiar with the study caution against drawing hasty conclusions because of the small number of patients involved, and because the therapy they received up to 30 years ago is crude by today's standards.
"Since then, there have been substantial refinements and improvements in radiotherapeutic techniques that have reduced the amount of healthy tissue that receives high doses of radiotherapy," Paul Brown and Jane Cerhan of the Mayo Clinic in Rochester, Minnesota wrote in a commentary.
While they say the findings are helpful, Brown and Cerhan call for further clinical trials to better understand the impact of radiation on the functioning of the mind.