WiIRE: the web interactive information retrieval experimentation system prototype
Information Processing and Management: an International Journal
TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing)
Methods for Evaluating Interactive Information Retrieval Systems with Users
Foundations and Trends in Information Retrieval
Usage based effectiveness measures: monitoring application performance in information retrieval
Proceedings of the 18th ACM conference on Information and knowledge management
Interactive information retrieval
Annual Review of Information Science and Technology
Hi-index | 0.00 |
Conducting Interactive Information Retrieval (IIR) research is often seen as an arduous and tedious process with a high barrier to entry. This high barrier is due to the overheads in developing and setting up even a simple lab-based IIR experiment. SCAMP (Search ConfigurAtor for experiMenting with PuppyIR) is a web-based tool that we have developed which enables researchers to configure standard IIR experiments. SCAMP provides the infrastructure that handles the major processes within the experimental flow (such as Participant Registration, Consent, Surveys, and the logging and tracking of tasks and participants through the experiment). Consequently, SCAMP reduces the time required to create an experiment and is ideal for undergraduate and masters students who would like to conduct an IIR experiment without extensive development. Furthermore, the tool is extensible. Other features can be easily added to SCAMP to customise the experiments. To evaluate SCAMP, we performed a within-subjects experiment where 48 participants used a web search engine with different search aids (query suggestion, spell correction, etc) to complete various web search tasks - to increase the difficulty of the search tasks certain query terms were banned. We use this evaluation to showcase the different features of the SCAMP system and report on how participants perform under difficult querying conditions.