Its all in the white matter in the brain, it seems. That crucial nerve cell-packed matter, effectively serving as the brain's wiring, degrades over time, eventually slowing down the cognitive process, say US researchers.
Harvard University used medical imaging techniques to compare the brains of 93 healthy people aged 18 to 93.
The scans showed the brain gradually loses the material it needs for one major region to communicate effectively with another.
The study, published in Neuron, suggests this slowly undermines sophisticated "higher" cognitive functions such as memory and learning.
This may help to explain why advanced age is often accompanied by a loss of mental agility, even in an otherwise healthy individual.
Lead researcher Jessica Andrews-Hanna said: "This research helps us to understand how and why our minds change as we get older, and why some individuals remain sharp into their 90s, while others' mental abilities decline as they age.
"One of the reasons for loss of mental ability may be that these systems in the brain are no longer in sync with one another."
Previous studies have focused on the effect of ageing on specific structures in the brain.
The latest work, using functional magnetic resonance imaging, was different because it examined the effect on communication between different regions.
The scans showed white matter degraded over time. In particular, they revealed a reduction in connections between the front and back regions of the brain.
As a result, while the younger brains were in sync, this was not always the case for older brains.
Older people whose brains remained in sync were more likely to perform better in a battery of tests of mental capacity than peers whose scans showed more evidence of disruption.
However, the pattern of disruption varied between individuals - as did their performance on individual tests.
The researchers found the system governing our internal thoughts, which tends to kick in when we are not focusing on processing information from the outside world, was particularly vulnerable to disruption.
The researchers said the study promises a better physiological understanding of cognitive decline, and may help research into the impact of risk factors such as heart disease.
Professor Randy Buckner, who worked on the study, said: "Understanding why we lose cognitive function as we age may help us to prolong our mental abilities later in life."
Professor Clive Ballard, of the Alzheimer's Society, said more work was needed to confirm and clarify the findings.
He said: "Understanding how the brain changes as people age is an important part of the fight to protect against cognitive diseases such as dementia.
"People displaying the signs of Alzheimer's disease were ruled out of the study, but those with subtle vascular changes in the brain may have been included.
"Further work is needed to establish if the pattern of change is related to age only, or to vascular changes in the brain."
Rebecca Wood, of the Alzheimer's Research Trust, said the research highlighted the complexity of the brain.
She said: "If we can better understand the normal effects of ageing on a brain then we can differentiate it from Alzheimer's and improve diagnosis."