How to improve the statistical power of the 10-fold cross validation scheme in recommender systems

Authors:
Andrej Košir;Ante Odić;Marko Tkalčič
Affiliations:
University of Ljubljana, Ljubljana, Slovenia;University of Ljubljana, Ljubljana, Slovenia;Johannes Kepler University, Linz, Austria
Venue:
Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation
Year:
2013

Citing 5
Cited 0

Evaluating collaborative filtering recommender systems

ACM Transactions on Information Systems (TOIS)
Statistical Comparisons of Classifiers over Multiple Data Sets

The Journal of Machine Learning Research
Matrix Factorization Techniques for Recommender Systems

Computer
Recommender Systems: An Introduction

Recommender Systems: An Introduction
A 3D approach to recommender system evaluation

Proceedings of the 2013 conference on Computer supported cooperative work companion

Quantified Score

Hi-index	0.00

Visualization

Abstract

At this stage development of recommender systems (RS), an evaluation of competing approaches (methods) yielding similar performances in terms of experiment reproduction is of crucial importance in order to direct the further development toward the most promising direction. These comparisons are usually based on the 10-fold cross validation scheme. Since the compared performances are often similar to each other, the application of statistical significance testing is inevitable in order to not to get misled by randomly caused differences of achieved performances. For the same reason, to reproduce experiments on a different set of experimental data, the most powerful significance testing should be applied. In this work we provide guidelines on how to achieve the highest power in the comparison of RS and we demonstrate them on a comparison of RS performances when different variables are contextualized.