People get vitamin D from sunlight and from oily fish like salmon, tuna or mackerel, as well as milk, eggs and cheese. It is also available in supplement form.
Reporting in the journal Neurology, international researchers found that people who were severely deficient in vitamin D were more than twice as likely to develop dementia and Alzheimer's disease as people who got enough.
The findings were based on a study of 1,658 adults aged 65 and over, who were healthy and able to walk without assistance.
The participants were followed for six years. By that point, 171 participants had developed dementia and 102 had Alzheimer's disease.
Those who were moderately deficient in vitamin D had a 53 percent increased risk of developing dementia of any kind. Those who were severely deficient saw their risk increase to 125 percent over those with adequate levels of vitamin D.
Similar numbers were noted for Alzheimer's disease: those who were moderately deficient were 69 percent more likely to develop this type of dementia, and the severely deficient were 122 percent more likely to get Alzheimer's.
"We expected to find an association between low vitamin D levels and the risk of dementia and Alzheimer's disease, but the results were surprising ?- we actually found that the association was twice as strong as we anticipated," said lead author David Llewellyn at the University of Exeter Medical School.
"Clinical trials are now needed to establish whether eating foods such as oily fish or taking vitamin D supplements can delay or even prevent the onset of Alzheimer's disease and dementia."
He added that the study stops short of showing whether or not vitamin D deficiency causes dementia, only that it shows there is a link that deserves further research.
Some 44 million people worldwide have dementia, a number that is expected to triple by 2050.
About one billion around the globe are thought to have low vitamin D levels.
The elderly can be particularly vulnerable to such a deficiency because their skin is less adept at converting sunlight into vitamin D.