Text-Mining Algorithm for Identifying Relevant Research on Chemicals and Disease for Public Database Developed

Researchers at North Carolina State University have developed a new text mining algorithm that can help identify the most relevant scientific research for a public database that reveals the effects of environmental chemicals on human health

The Comparative Toxicogenomics Database (CTD), managed in part by the lead authors, is a manually curated, public database that correlates environmental chemicals with their effects on genes and human health. Thousands of new research papers are published each day, and finding the most relevant ones to include can be challenging. As Davis explains, "Over 33,000 scientific papers have been published on heavy metal toxicity alone, going as far back as 1926. We simply can't read and code them all. And, with the help of this new algorithm, we don't have to."

The algorithm described in the study assigns scientific articles a score based on data content, biological and toxicological relevance and several other parameters. Integrating this algorithm with the current system of manual curation helped the researchers significantly improve their process by prioritizing more relevant articles for inclusion in the database, increasing productivity by 27 percent and novel data content by 100 percent.

Only 15 percent of the papers studied were incorrectly identified by the algorithm as being highly relevant, but the researchers were able to identify the reasons for these inaccurate results. "Now, we can go back and tweak the algorithm to account for this and fine-tune the system," says Wiegers.

"We're not at the point yet where a computer can read and extract all the relevant data on its own," concludes Davis, "but having this text-mining process to direct us toward the most informative articles is a huge first step."

Source: Eurekalert