The use of MMR, diversity-based reranking for reordering documents and producing summaries
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Evaluating collaborative filtering recommender systems
ACM Transactions on Information Systems (TOIS)
Improving recommendation lists through topic diversification
WWW '05 Proceedings of the 14th international conference on World Wide Web
Being accurate is not enough: how accuracy metrics have hurt recommender systems
CHI '06 Extended Abstracts on Human Factors in Computing Systems
Learning diverse rankings with multi-armed bandits
Proceedings of the 25th international conference on Machine learning
Novelty and diversity in information retrieval evaluation
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Rank-biased precision for measurement of retrieval effectiveness
ACM Transactions on Information Systems (TOIS)
CARD: a decision-guidance framework and application for recommending composite alternatives
Proceedings of the 2008 ACM conference on Recommender systems
A new approach to evaluating novel recommendations
Proceedings of the 2008 ACM conference on Recommender systems
Proceedings of the Second ACM International Conference on Web Search and Data Mining
Expected reciprocal rank for graded relevance
Proceedings of the 18th ACM conference on Information and knowledge management
Temporal diversity in recommender systems
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
A study of heterogeneity in recommendations for a social music service
Proceedings of the 1st International Workshop on Information Heterogeneity and Fusion in Recommender Systems
A comparative analysis of cascade measures for novelty and diversity
Proceedings of the fourth ACM international conference on Web search and data mining
Characterizing search intent diversity into click models
Proceedings of the 20th international conference on World wide web
System effectiveness, user models, and user utility: a conceptual framework for investigation
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Intent-oriented diversity in recommender systems
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Empirical analysis of predictive algorithms for collaborative filtering
UAI'98 Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence
Improving Aggregate Recommendation Diversity Using Ranking-Based Techniques
IEEE Transactions on Knowledge and Data Engineering
Workshop on novelty and diversity in recommender systems - DiveRS 2011
Proceedings of the fifth ACM conference on Recommender systems
Auralist: introducing serendipity into music recommendation
Proceedings of the fifth ACM international conference on Web search and data mining
Proceedings of the 14th Annual International Conference on Electronic Commerce
Explicit relevance models in intent-oriented information retrieval diversification
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Adaptive diversification of recommendation results via latent factor portfolio
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Personalized diversification of search results
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Pareto-efficient hybridization for multi-objective recommender systems
Proceedings of the sixth ACM conference on Recommender systems
Case study on the business value impact of personalized recommendations on a large online retailer
Proceedings of the sixth ACM conference on Recommender systems
Workshop on recommendation utility evaluation: beyond RMSE -- RUE 2012
Proceedings of the sixth ACM conference on Recommender systems
Exploiting relevance, novelty and diversity in tag recommendation
Proceedings of the 18th Brazilian symposium on Multimedia and the web
A comparative study of heterogeneous item recommendations in social systems
Information Sciences: an International Journal
Building user profiles to improve user experience in recommender systems
Proceedings of the sixth ACM international conference on Web search and data mining
An analysis of topical proximity in the twitter social graph
SocInfo'12 Proceedings of the 4th international conference on Social Informatics
Serendipitous Personalized Ranking for Top-N Recommendation
WI-IAT '12 Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01
Exploiting novelty and diversity in tag recommendation
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
Knowledge-Based Systems
A general evaluation measure for document organization tasks
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Beyond relevance: on novelty and diversity in tag recommendation
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
A new collaborative filtering approach for increasing the aggregate diversity of recommender systems
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Big & personal: data and models behind netflix recommendations
Proceedings of the 2nd International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications
Automatically recommending multimedia content for use in group reminiscence therap
Proceedings of the 1st ACM international workshop on Multimedia indexing and information retrieval for healthcare
Understanding and improving relational matrix factorization in recommender systems
Proceedings of the 7th ACM conference on Recommender systems
Exploiting non-content preference attributes through hybrid recommendation method
Proceedings of the 7th ACM conference on Recommender systems
Topic diversity in tag recommendation
Proceedings of the 7th ACM conference on Recommender systems
Personalised ranking with diversity
Proceedings of the 7th ACM conference on Recommender systems
Bridging memory-based collaborative filtering and text retrieval
Information Retrieval
Multi-objective mobile app recommendation: A system-level collaboration approach
Computers and Electrical Engineering
Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols
User Modeling and User-Adapted Interaction
Hi-index | 0.00 |
The Recommender Systems community is paying increasing attention to novelty and diversity as key qualities beyond accuracy in real recommendation scenarios. Despite the raise of interest and work on the topic in recent years, we find that a clear common methodological and conceptual ground for the evaluation of these dimensions is still to be consolidated. Different evaluation metrics have been reported in the literature but the precise relation, distinction or equivalence between them has not been explicitly studied. Furthermore, the metrics reported so far miss important properties such as taking into consideration the ranking of recommended items, or whether items are relevant or not, when assessing the novelty and diversity of recommendations. We present a formal framework for the definition of novelty and diversity metrics that unifies and generalizes several state of the art metrics. We identify three essential ground concepts at the roots of novelty and diversity: choice, discovery and relevance, upon which the framework is built. Item rank and relevance are introduced through a probabilistic recommendation browsing model, building upon the same three basic concepts. Based on the combination of ground elements, and the assumptions of the browsing model, different metrics and variants unfold. We report experimental observations which validate and illustrate the properties of the proposed metrics.