GroupLens: applying collaborative filtering to Usenet news
Communications of the ACM
Amazon.com Recommendations: Item-to-Item Collaborative Filtering
IEEE Internet Computing
Is seeing believing?: how recommender system interfaces affect users' opinions
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Evaluating collaborative filtering recommender systems
ACM Transactions on Information Systems (TOIS)
Item-based top-N recommendation algorithms
ACM Transactions on Information Systems (TOIS)
Being accurate is not enough: how accuracy metrics have hurt recommender systems
CHI '06 Extended Abstracts on Human Factors in Computing Systems
The Google Similarity Distance
IEEE Transactions on Knowledge and Data Engineering
Lessons from the Netflix prize challenge
ACM SIGKDD Explorations Newsletter - Special issue on visual analytics
CARD: a decision-guidance framework and application for recommending composite alternatives
Proceedings of the 2008 ACM conference on Recommender systems
Personalized news recommendation based on click behavior
Proceedings of the 15th international conference on Intelligent user interfaces
Content-based recommendation systems
The adaptive web
RecLab: a system for eCommerce recommender research with real data, context and feedback
Proceedings of the 2011 Workshop on Context-awareness in Retrieval and Recommendation
High quality recommendations for small communities: the case of a regional parent network
Proceedings of the sixth ACM conference on Recommender systems
Case study on the business value impact of personalized recommendations on a large online retailer
Proceedings of the sixth ACM conference on Recommender systems
A live comparison of methods for personalized article recommendation at forbes.com
ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part II
A group recommender for movies based on content similarity and popularity
Information Processing and Management: an International Journal
| Hi-index | 0.00 |
Evaluation have been an important subject since the early days of recommender systems. In online test, the click-through rate (CTR) is often adopted as the metric. However, recommended items with higher CTR does not imply higher relevance of two items since factors like item popularity or item serendipity may influence user's click behavior. We argue that the relevance of recommendation system is also desirable in many real applications. Here relevant means relevance in a human perceptible way. Relevant recommendations not only increase the users' trust to the system but are extremely useful for the vast number of anonymous user as their recommendations may only be made based on the current item. In this paper, we empirically examine the relation between the relevance of recommendations and the corresponding CTR with a few representative ItemCF algorithms through online data from a TV show/movie website, Hulu. Experiments show that algorithms with higher overall CTR may not correspond to higher relevance. Thus CTR may not be the optimal metric for online evaluation of recommender systems if producing relevant recommendations is of importance.