Robustness of adaptive filtering methods in a cross-benchmark evaluation

Authors:
Yiming Yang;Shinjae Yoo;Jian Zhang;Bryan Kisiel
Affiliations:
Carnegie Mellon University, Pittsburgh, PA;Carnegie Mellon University, Pittsburgh, PA;Carnegie Mellon University, Pittsburgh, PA;Carnegie Mellon University, Pittsburgh, PA
Venue:
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
Year:
2005

Citing 9
Cited 19

Incremental relevance feedback for information filtering

SIGIR '96 Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval
Boosting and Rocchio applied to text filtering

Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Learning while filtering documents

Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Maximum likelihood estimation for filtering thresholds

Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Text Categorization Based on Regularized Linear Classification Methods

Information Retrieval
Topic detection and tracking evaluation overview

Topic detection and tracking
Robustness of regularized linear classification methods in text categorization

Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Margin-based local regression for adaptive filtering

CIKM '03 Proceedings of the twelfth international conference on Information and knowledge management
Using bayesian priors to combine classifiers for adaptive filtering

Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

Utility-based information distillation over temporally sequenced documents

SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Efficient bayesian hierarchical user modeling for recommendation system

SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Generalizing from relevance feedback using named entity wildcards

Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
An evaluation of adaptive filtering in the context of realistic task-based information exploration

Information Processing and Management: an International Journal
Toward incorporating a task-stage identification technique into the long-term document support process

Information Processing and Management: an International Journal
Towards modeling threaded discussions using induced ontology knowledge

AAAI'06 proceedings of the 21st national conference on Artificial intelligence - Volume 2
Immune Learning in a Dynamic Information Environment

ICARIS '09 Proceedings of the 8th International Conference on Artificial Immune Systems
RADAR: a personal assistant that learns to reduce email overload

AAAI'08 Proceedings of the 23rd national conference on Artificial intelligence - Volume 3
A demonstration of the RADAR personal assistant

AAAI'08 Proceedings of the 23rd national conference on Artificial intelligence - Volume 3
Learning dynamic information needs: A collaborative topic variation inspection approach

Journal of the American Society for Information Science and Technology
Agent-assisted task management that reduces email overload

Proceedings of the 15th international conference on Intelligent user interfaces
Overview of CLEF 2008 INFILE pilot track

CLEF'08 Proceedings of the 9th Cross-language evaluation forum conference on Evaluating systems for multilingual and multimodal information access
Selected new training documents to update user profile

CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Information filtering evaluation: overview of CLEF 2009 INFILE track

CLEF'09 Proceedings of the 10th cross-language evaluation forum conference on Multilingual information access evaluation: text retrieval experiments
A pattern mining approach for information filtering systems

Information Retrieval
Filtering semi-structured documents based on faceted feedback

Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Modeling personalized email prioritization: classification-based and regression-based approaches

Proceedings of the 20th ACM international conference on Information and knowledge management
Indices of novelty for emerging topic detection

Information Processing and Management: an International Journal
On sparsity and drift for effective real-time filtering in microblogs

Proceedings of the 22nd ACM international conference on Conference on information & knowledge management

Quantified Score

Hi-index	0.02

Visualization

Abstract

This paper reports a cross-benchmark evaluation of regularized logistic regression (LR) and incremental Rocchio for adaptive filtering. Using four corpora from the Topic Detection and Tracking (TDT) forum and the Text Retrieval Conferences (TREC) we evaluated these methods with non-stationary topics at various granularity levels, and measured performance with different utility settings. We found that LR performs strongly and robustly in optimizing T11SU (a TREC utility function) while Rocchio is better for optimizing Ctrk (the TDT tracking cost), a high-recall oriented objective function. Using systematic cross-corpus parameter optimization with both methods, we obtained the best results ever reported on TDT5, TREC10 and TREC11. Relevance feedback on a small portion (0.05~0.2%) of the TDT5 test documents yielded significant performance improvements, measuring up to a 54% reduction in Ctrk and a 20.9% increase in T11SU (with b=0.1), compared to the results of the top-performing system in TDT2004 without relevance feedback information.