Not only is the BRCA1 mutation rare in the overall female population, they said, but it is also expensive to test for at a US laboratory that controversially claims patent rights to the gene.
"We don't want everybody to go out and say, 'I want this test'," Otis Brawley, chief medical officer of the American Cancer Society, told AFP in a telephone interview.
"We want people to think about what their family history is, and to talk to their doctor" who might refer them to a genetic counselor to determine if testing is necessary, Brawley said.
"Women who have first-degree relatives -- that is a sister or a mother -- who had breast or ovarian cancer at a young age should especially have these conversations," he said.
"All women should not be tested for this gene. It is not that common," said Los Angeles-based surgeon, author, breast cancer researcher and leukemia survivor Susan Love.
"The people who should consider being tested are the ones who have had a lot of breast or ovarian cancer in their families," she told CNN.
In Jolie's case, the mother of the 37-year-old Oscar-winning actress and UN humanitarian activist -- star of such action films as "Salt" and "Lara Croft: Tomb Raider" -- died of ovarian cancer at the age of 56.
Writing on the op-ed page of The New York Times on the eve of the Cannes film festival, where she is a red-carpet favorite, Jolie said genetic tests had established that she was carrying a "faulty" gene known as BRCA1.
"She's very brave to speak out," said Katherine McLane, vice president for communications at Livestrong, the cancer charity founded by cyclist and cancer survivor Lance Armstrong before his fall from grace in a doping scandal.
But in a telephone interview with AFP, McLane voiced hope that Jolie's experience would also generate "more attention about affordable health care for all women."
In a little-noted part of her op-ed, Jolie, one of Hollywood's highest-paid stars, decried the fact that a BRCA1 test -- which costs more than $3,000 in the United States -- is out of reach for many women around the world.
Myriad Genetics is the only US lab that offers the test, and it is at the heart of a US Supreme Court battle over its claim of patent rights to BRCA1 and BRCA2. A ruling is expected in June.
On the NASDAQ stock market Tuesday, shares in Myriad Genetics leaped four percent on news of Jolie's article to $34.45, its best level since mid-2009, before easing back.
"We strongly believe appropriate use of many of our diagnostic tests can help reduce illness, hospitalizations and other costly interventions, and potentially lower health care costs," Myriad Genetics said in a statement.
It declined to say whether it was the lab that tested Jolie. Several facilities in Europe offer the same test, according to the American Civil Liberties Union, which is challenging Myriad Genetics' patent claim in court.
Overall, cancer death rates in the United States have been declining among both men and women since the early 1990s, according to the National Cancer Institute, a US government agency.
But while 12 percent of all US women will develop breast cancer at some point, the chances are five times greater for the few with a BRCA1 or BRCA2 mutation, it says on its website (www.cancer.gov).
Both mutations can also turn up in men, ramping up their risk of breast, pancreatic, testicular and prostate cancer.