User-centred versus system-centred evaluation of a personalization system

  • Authors:

  • Alberto Díaz;Antonio García;Pablo Gervás

  • Affiliations:

  • Dep. Ingeniería del Software e Inteligencia Artificial, Facultad de Informática - Universidad Complutense de Madrid, c/ Profesor José García Santesmases, s/n, Madrid 28040, Spa ...;Dep. Ciencias de la Comunicación I, Facultad de Ciencias de la Comunicación - Universidad Rey Juan Carlos, Camino del Molino, s/n, Fuenlabrada, Madrid 28943, Spain;Dep. Ingeniería del Software e Inteligencia Artificial, Facultad de Informática - Universidad Complutense de Madrid, c/ Profesor José García Santesmases, s/n, Madrid 28040, Spa ...

  • Venue:
  • Information Processing and Management: an International Journal
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

Some of the most popular measures to evaluate information filtering systems are usually independent of the users because they are based in relevance judgments obtained from experts. On the other hand, the user-centred evaluation allows showing the different impressions that the users have perceived about the system running. This work is focused on discussing the problem of user-centred versus system-centred evaluation of a Web content personalization system where the personalization is based on a user model that stores long term (section, categories and keywords) and short term interests (adapted from user provided feedback). The user-centred evaluation is based on questionnaires filled in by the users before and after using the system and the system-centred evaluation is based on the comparison between ranking of documents, obtained from the application of a multi-tier selection process, and binary relevance judgments collected previously from real users. The user-centred and system-centred evaluations performed with 106 users during 14 working days have provided valuable data concerning the behaviour of the users with respect to issues such as document relevance or the relative importance attributed to different ways of personalization. The results obtained shows general satisfaction on both the personalization processes (selection, adaptation and presentation) and the system as a whole.