Smoking rots your brain by damaging memory, learning and reasoning, warn researchers.
A study of 8,800 people over 50 showed high blood pressure and being overweight also seemed to affect the brain, but to a lesser extent.
The researchers were investigating links between the likelihood of a heart attack or stroke and the state of the brain.
Data about the health and lifestyle of a group of over-50s was collected and brain tests, such as making participants learn new words or name as many animals as they could in a minute, were also performed. They were all tested again after four and then eight years.
They found that the overall risk of a heart attack or stroke was "significantly associated with cognitive decline" with those at the highest risk showing the greatest decline.
It also said there was a "consistent association" between smoking and lower scores in the tests.
"Cognitive decline becomes more common with ageing and for an increasing number of people interferes with daily functioning and well-being," the BBC quoted Dr Alex Dregan, one of the researchers, as saying
"We have identified a number of risk factors which could be associated with accelerated cognitive decline, all of which, could be modifiable."
"We need to make people aware of the need to do some lifestyle changes because of the risk of cognitive decline," he added.
The researchers do not know how such a decline could affect people going about their daily life. They are also unsure whether the early drop in brain function could lead to conditions such as dementia.
Dr Simon Ridley, from Alzheimer's Research UK, said: "Research has repeatedly linked smoking and high blood pressure to a greater risk of cognitive decline and dementia, and this study adds further weight to that evidence.
"Cognitive decline as we age can develop into dementia, and unravelling the factors that are linked to this decline could be crucial for finding ways to prevent the condition. These results underline the importance of looking after your cardiovascular health from mid-life," he noted.
The Alzheimer's Society said: "We all know smoking, a high blood pressure, high cholesterol levels and a high BMI [Body Mass Index] is bad for our heart. This research adds to the huge amount of evidence that also suggests they can be bad for our head too.
"One in three people over 65 will develop dementia but there are things people can do to reduce their risk. Eating a balanced diet, maintaining a healthy weight, exercising regularly, getting your blood pressure and cholesterol checked and not smoking can all make a difference," it suggested.
The findings were published in the journal Age and Ageing.