The scientists, funded by the Medical Research Council and Wellcome Trust, have discovered that people with OCD and their close family members show under-activation of brain areas responsible for stopping habitual behaviour.
The study is published in the 18 July edition of Science.
Patients with OCD suffer from recurrent intrusive thoughts (obsessions) that are distressing and hard to suppress.
Examples include fears of contamination, or that something terrible will happen to a loved one. They also suffer from repetitive rituals (compulsions), which are often designed to neutralise these thoughts. Examples include hand-washing and checking gas hobs.
Dr Samuel Chamberlain at the University of Cambridge's Department of Psychiatry used functional magnetic resonance imaging (fMRI) to measure brain activity in the lateral orbitofrontal cortex (OFC). Located in the frontal lobes the lateral OFC is involved in decision-making and behaviour.
Volunteers were asked to look at two pictures on a screen, each image had a house and a face superimposed.
The volunteers were asked to use trial and error to work out whether the house or face was the correct target. Volunteers pressed a button to indicate which image they believed to be the target and feedback of 'correct' or 'incorrect' was given on the screen.
After the correct target had been identified six times in a row it changed so the volunteer had to learn again. fMRI was used to monitor their patterns of brain activity throughout.
Fourteen volunteers without a family history of OCD, 14 people with OCD and 12 immediate relatives of these patients took the picture test. Later comparison of fMRI images of their brain activity throughout showed under-activation in the lateral orbitofrontal cortex and other brain areas in both the OCD patients and their family members.
Dr Chamberlain said: "Impaired function in brain areas controlling flexible behaviour probably predisposes people to developing the compulsive rigid symptoms that are characteristic of OCD.
"This study shows that these brain changes run in families and represent a candidate vulnerability factor. The current diagnosis of OCD is subjective and improved understanding of the underlying causes of OCD could lead to more accurate diagnosis and improved clinical treatments."