Evaluating and optimizing autonomous text classification systems
SIGIR '95 Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval
Assessing agreement on classification tasks: the kappa statistic
Computational Linguistics
Using asymmetric distributions to improve text classifier probability estimates
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Learning and evaluating classifiers under sample selection bias
ICML '04 Proceedings of the twenty-first international conference on Machine learning
Optimized stratified sampling for approximate query processing
ACM Transactions on Database Systems (TODS)
Online supervised spam filter evaluation
ACM Transactions on Information Systems (TOIS)
A simple and efficient sampling method for estimating AP and NDCG
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
A stratified traffic sampling methodology for seeing the big picture
Computer Networks: The International Journal of Computer and Telecommunications Networking
Threshold selection for web-page classification with highly skewed class distribution
Proceedings of the 18th international conference on World wide web
Toward interactive training and evaluation
Proceedings of the 20th ACM international conference on Information and knowledge management
| Hi-index | 0.00 |
Deploying a classifier to large-scale systems such as the web requires careful feature design and performance evaluation. Evaluation is particularly challenging because these large collections frequently change. In this paper we adapt stratified sampling techniques to evaluate the precision of classifiers deployed in large-scale systems. We investigate different types of stratification strategies, and then we derive a new online sampling algorithm that incrementally approximates the theoretical optimal disproportionate sampling strategy. In experiments, the proposed algorithm significantly outperforms both simple random sampling as well as other types of stratified sampling, with an average reduction of about 20% in labeling effort to reach the same confidence and interval-bounds on precision