A comparison of evaluation metrics for biomedical journals, articles, and websites in terms of sensitivity to topic

Authors:
Lawrence D. Fu;Yindalon Aphinyanaphongs;Lily Wang;Constantin F. Aliferis
Affiliations:
New York University Medical Center, Center for Health Informatics and Bioinformatics, 227 E 30th St., 7th Floor, New York, NY 10016, USA;New York University Medical Center, Center for Health Informatics and Bioinformatics, 227 E 30th St., 7th Floor, New York, NY 10016, USA;Department of Biostatistics, Vanderbilt University School of Medicine, S-2323 Medical Center North, Nashville, TN 37232, USA;New York University Medical Center, Center for Health Informatics and Bioinformatics, 227 E 30th St., 7th Floor, New York, NY 10016, USA
Venue:
Journal of Biomedical Informatics
Year:
2011

Citing 8
Cited 0

The anatomy of a large-scale hypertextual Web search engine

WWW7 Proceedings of the seventh international conference on World Wide Web 7
Finding authorities and hubs from link structures on the World Wide Web

Proceedings of the 10th international conference on World Wide Web
Stable algorithms for link analysis

Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
A Tutorial on Support Vector Machines for Pattern Recognition

Data Mining and Knowledge Discovery
Topic-Sensitive PageRank: A Context-Sensitive Ranking Algorithm for Web Search

IEEE Transactions on Knowledge and Data Engineering
Rank-Stability and Rank-Similarity of Link-Based Web Ranking Algorithms in Authority-Connected Graphs

Information Retrieval
Topical link analysis for web search

SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Using Multivariate Statistics (5th Edition)

Using Multivariate Statistics (5th Edition)

Quantified Score

Hi-index	0.00

Visualization

Abstract

Evaluating the biomedical literature and health-related websites for quality are challenging information retrieval tasks. Current commonly used methods include impact factor for journals, PubMed's clinical query filters and machine learning-based filter models for articles, and PageRank for websites. Previous work has focused on the average performance of these methods without considering the topic, and it is unknown how performance varies for specific topics or focused searches. Clinicians, researchers, and users should be aware when expected performance is not achieved for specific topics. The present work analyzes the behavior of these methods for a variety of topics. Impact factor, clinical query filters, and PageRank vary widely across different topics while a topic-specific impact factor and machine learning-based filter models are more stable. The results demonstrate that a method may perform excellently on average but struggle when used on a number of narrower topics. Topic-adjusted metrics and other topic robust methods have an advantage in such situations. Users of traditional topic-sensitive metrics should be aware of their limitations.