In situ evaluation of recommender systems: Framework and instrumentation

  • Authors:

  • M. Funk;A. Rozinat;E. Karapanos;A. K. Alves de Medeiros;A. Koca

  • Affiliations:

  • Faculty of Electrical Engineering, Eindhoven University of Technology, The Netherlands;Department of Information Systems, Eindhoven University of Technology, The Netherlands;Faculty of Industrial Design, Eindhoven University of Technology, The Netherlands;Department of Information Systems, Eindhoven University of Technology, The Netherlands;Faculty of Industrial Design, Eindhoven University of Technology, The Netherlands

  • Venue:
  • International Journal of Human-Computer Studies
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper deals with the evaluation of the recommendation functionality inside a connected consumer electronics product in prototype stage. This evaluation is supported by a framework to access and analyze data about product usage and user experience. The strengths of this framework lie in the collection of both objective data (i.e., ''What is the user doing with the product?'') and subjective data (i.e., ''How is the user experiencing the product?''), which are linked together and analyzed in a combined way. The analysis of objective data provides insights into how the system is actually used in the field. Combined with the subjective data, personal opinions and evaluative judgments on the product quality can be then related to actual user behavior. In order to collect these data in a most natural context, remote data collection allows for extensive user testing within habitual environments. We have applied our framework to the case of an interactive TV recommender system application to illustrate that the user experience of recommender systems can be evaluated in real-life usage scenarios.