The simple game, which is based on the type of prey favoured by the predatory bluegill sunfish, featured red dots that moved and swarmed in different ways against a translucent screen.
When they projected the game into a fish tank, they found that the fish were less likely to try to attack the dots when they moved in a group formation.
"By creating an immersive video game for the fish we were able to have complete control over the parameters," senior researcher Dr Iain Couzin from the Department of Ecology and Evolutionary Biology at Princeton University told BBC News.
The size and colour of the simple prey graphics were carefully designed, he said.
He said it was important that the game had been coded so that the movement of the dots did not become predictable.
"In any computer game if you have one type of enemy it's easy to learn. It would be fascinating to understand whether the fish learned to play the game better over time," said Couzin.
The team is now looking at using 3D technology to create a more photorealistic world in which to study fish behaviour.
"We're developing an automating tracking system so we can track the position of their eyes and reconstruct a virtual world of prey items, using conventional projectors. It will be a fully 3D virtual world to these organisms," said Couzin.
The research has been published in the Science journal.