CROC: A New Evaluation Criterion for Recommender Systems

Authors:
Andrew I. Schein;Alexandrin Popescul;Lyle H. Ungar;David M. Pennock
Affiliations:
University of Pennsylvania, Department of Computer and Information Science, Levine Hall, 3330 Walnut Street, Philadelphia, PA 19104-6389, USA ais@cis.upenn.edu;University of Pennsylvania, Department of Computer and Information Science, Levine Hall, 3330 Walnut Street, Philadelphia, PA 19104-6389, USA popescul@cis.upenn.edu;University of Pennsylvania, Department of Computer and Information Science, Levine Hall, 3330 Walnut Street, Philadelphia, PA 19104-6389, USA ungar@cis.upenn.edu;Overture Services, Inc., 74 N. Pasadena Ave, 3rd floor, Pasadena, CA 91103, USA david.pennock@overture.com
Venue:
Electronic Commerce Research
Year:
2005

Citing 25
Cited 9

GroupLens: an open architecture for collaborative filtering of netnews

CSCW '94 Proceedings of the 1994 ACM conference on Computer supported cooperative work
Recommending and evaluating choices in a virtual community of use

CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Social information filtering: algorithms for automating “word of mouth”

CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Recommender systems

Communications of the ACM
GroupLens: applying collaborative filtering to Usenet news

Communications of the ACM
Recommendation as classification: using social and content-based information in recommendation

AAAI '98/IAAI '98 Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence
Probabilistic latent semantic indexing

Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval
An algorithmic framework for performing collaborative filtering

Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval
Combining collaborative filtering with personal agents for better recommendations

AAAI '99/IAAI '99 Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence
Content-based book recommending using learning for text categorization

DL '00 Proceedings of the fifth ACM conference on Digital libraries
Analysis of recommendation algorithms for e-commerce

Proceedings of the 2nd ACM conference on Electronic commerce
Item-based collaborative filtering recommendation algorithms

Proceedings of the 10th international conference on World Wide Web
Methods and metrics for cold-start recommendations

SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Introduction to Modern Information Retrieval

Introduction to Modern Information Retrieval
Collaborative Filtering Using Weighted Majority Prediction Algorithms

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Learning Collaborative Information Filters

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Latent Class Models for Collaborative Filtering

IJCAI '99 Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence
Towards a More Comprehensive Comparison of Collaborative Filtering Algorithms

SPIRE 2002 Proceedings of the 9th International Symposium on String Processing and Information Retrieval
Dependency Networks for Collaborative Filtering and Data Visualization

UAI '00 Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence
Collaborative Filtering by Personality Diagnosis: A Hybrid Memory and Model-Based Approach

UAI '00 Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence
Probabilistic Models for Unified Collaborative and Content-Based Recommendation in Sparse-Data Environments

UAI '01 Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence
Social Choice Theory and Recommender Systems: Analysis of the Axiomatic Foundations of Collaborative Filtering

Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence
An Efficient Boosting Algorithm for Combining Preferences

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Learning to order things

Journal of Artificial Intelligence Research
Empirical analysis of predictive algorithms for collaborative filtering

UAI'98 Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence

Utilizing Popularity Characteristics for Product Recommendation

International Journal of Electronic Commerce
Analysis and Classification of Multi-Criteria Recommender Systems

World Wide Web
A new similarity measure for collaborative filtering to alleviate the new user cold-starting problem

Information Sciences: an International Journal
An empirical study of a cross-level association rule mining approach to cold-start recommendations

Knowledge-Based Systems
MUADDIB: A distributed recommender system supporting device adaptivity

ACM Transactions on Information Systems (TOIS)
Evaluating customer aid functions of online stores with agent-based models of customer behavior and evolution strategy

Information Sciences: an International Journal
Recommender system architecture for adaptive green marketing

Expert Systems with Applications: An International Journal
Recommending multimedia web services in a multi-device environment

Information Systems
Multi-agent technology and ontologies to support personalization in B2C E-Commerce

Electronic Commerce Research and Applications

Quantified Score

Hi-index	0.00

Visualization

Abstract

Evaluation of a recommender system algorithm is a challenging task due to the many possible scenarios in which such systems may be deployed. We have designed a new performance plot called the CROC curve with an associated statistic: the area under the curve. Our CROC curve supplements the widely used ROC curve in recommender system evaluation by discovering performance characteristics that standard ROC evaluation often ignores. Empirical studies on two domains and including several recommender system algorithms demonstrate that combining ROC and CROC curves in evaluation can lead to a more informed characterization of performance than using either curve alone.