Two proteins - called Rac and Rho - are responsible for the shape switch, Institute of Cancer researchers said in the journal Cell.
Knowledge of how a cancer spreads could improve treatments, experts said.
The spread of cancer cells from the initial tumour to other sites of the body, a process known as metastasis, is one of the biggest problems in treatment.
Melanoma, the most dangerous form of skin cancer, is particularly aggressive.
Study leader, Professor Chris Marshall, said his team had been able to view cells in live tumours rapidly undergoing these changes in shape.
They discovered that when Rac is switched on it encourages the cell to become elongated and simultaneously suppresses the activity of the competing Rho protein.
Conversely, when cells adopt the round form, a protein activated by Rho switches off Rac activation.
It is thought these alternate shapes may enable tumour cells to deal with different situations during cancer spread.
Tests suggest that a round-shaped tumour cell may be better equipped to survive in the bloodstream than elongated cells.
Professor Marshall said: "The research has found the constant competition between two proteins called Rac and Rho is responsible for allowing the cancer cells to change shape and spread through the body.
"By explaining a key part of that process, our research brings new hope for future therapies to fight cancer."
Co-author, Dr Victoria Sanz-Moreno, said: "Until now the conversion between different types of movement of individual cancer cells had been observed but the key players had not been identified.
"We are excited to discover that the amount and the activity of these proteins in the tumour cell regulates its shape and the mechanism for it to move and invade surrounding tissue."
Dr Lesley Walker, Cancer Research UK director of cancer information, said: "Successful treatment tends to be much more difficult if the cancer has spread.
"This exciting study has shed light on some of the key molecules involved in the signalling pathways that encourage cells to move around the body.
"Knowing more about how cancer spreads will hopefully lead to the identification of new drug targets which will enable scientists to develop anti-cancer drugs to block these pathways."