The participants were held as they contested the "Miss Universe Asia 2008" title at a resort hotel in the northeastern state of Kelantan, which is ruled by the Pan-Malaysia Islamic Party (PAS), the New Sunday Times said.
The PAS wants to turn Malaysia into a theocratic state under Islamic rule, and has made headlines for ordering fines for women wearing skimpy clothes and stricter enforcement of laws on separate male and female queues in shops.
Mohamad Abdul Aziz, chief assistant director of enforcement in the state, told the paper that 16 people aged between 20 and 30 were detained.
Some were dressed in Malay traditional outfits and others in evening gowns, the paper added.
Mohamad said another group of 50 transvestites who were preparing to join the competition managed to escape arrest, while about 300 people were present at the hotel to watch the event.
The Sunday Times said most contestants were teachers and bank employees.
Mohamad said action would be taken against the transvestites under Islamic Sharia law "for dressing and acting like a woman."
They could be fined or jailed for six months if found guilty, he said.