1University Information Technology Services, Indiana University, Indianapolis, IN, 46202, USA
2Indiana University School of Informatics, Purdue University School of Science, Dept. of Computer and Information Science, Indianapolis, IN, 46202, USA. E-mail: jakechen@iupui.edu
Biomedical research publications contain a large amount of biomedical knowledge that is not extracted into biological databases. Existing computerized systems such as PubMed facilitate automated document retrieval by indexing terms from a relatively short list of keywords or from full document text. Such techniques are found to suffer from poor recall or precision, and are not sensitive to user queries within the context of a specific disease. We developed several methods and incorporated them into a web application to demonstrate how to improve precision and recall of document retrieval. First, we collected and normalized frequency distributions of observed biomedical terms by sampling from 1,500,000 MEDLINE abstracts. Then, we used these term statistics to derive a p-value for each term that occurs in a biomedical document. Each document (article or abstract) could then be summarized into a “gist”, which consists of a list of automatically curated keywords with significant p-values. The gist was then used to help improve the precision of document retrieval. Next, we integrated three external biological data sources as three separate thesauri to expand query search terms as a way to further improve document recall. In the end, we incorporated the above methods and results into a web-based system (http://discover.uits.indiana.edu:8340/cgi-bin/TextMining/search_p.pl) built on the Oracle 10g text mining platform. The term statistics, document “gists”, and query expansion techniques from our project can be readily shared with other biomedical text mining projects. Our software system also provides a reference model for future implementations of a biomedical text management system with all PubMed texts.
