Information Retrieval Experiment
Information Retrieval Experiment
TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing)
Consumer reactions to electronic shopping on the world wide web
International Journal of Electronic Commerce
Some(what) grand challenges for information retrieval
ACM SIGIR Forum
To swing or not to swing: learning when (not) to advertise
Proceedings of the 17th ACM conference on Information and knowledge management
Controlled experiments on the web: survey and practical guide
Data Mining and Knowledge Discovery
Methods for Evaluating Interactive Information Retrieval Systems with Users
Foundations and Trends in Information Retrieval
Report on the SIGIR 2009 workshop on the future of IR evaluation
ACM SIGIR Forum
A private living lab for requirements based evaluation
Proceedings of the 2013 workshop on Living labs for information retrieval evaluation
| Hi-index | 0.00 |
The notion of having a "living lab" to undertaken evaluations has been proposed by a number of proponents within the field of Information Retrieval (IR). However, what such a living lab might look like and how it might be setup has not been discussed in detail. Living labs have a number of appealing points such as realistic evaluation contexts where tasks are directly linked to user experience and the closer integration of research/academia and development/industry facilitating more efficient knowledge transfer. However, operationalizing a living lab opens up a number of concerns regarding security, privacy, etc. as well as challenges regarding the design, development and maintenance of the infrastructure required to support such evaluations. Here, we aim to further the discussion on living labs for IR evaluation and propose one possible architecture to create such an evaluation environment. To focus discussion, we put forward a proposal for a living lab on product search tasks within the context of an online shop.