The pragmatics of information retrieval experimentation, revisited
Information Processing and Management: an International Journal - Special issue on evaluation issues in information retrieval
Measuring usability: preference vs. performance
Communications of the ACM
Self disclosure on computer forms: meta-analysis and implications
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
WiIRE: the web interactive information retrieval experimentation system prototype
Information Processing and Management: an International Journal
IIiX Proceedings of the 1st international conference on Information interaction in context
Evaluation by comparing result sets in context
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Effects of performance feedback on users' evaluations of an interactive IR system
Proceedings of the second international symposium on Information interaction in context
Methods for Evaluating Interactive Information Retrieval Systems with Users
Foundations and Trends in Information Retrieval
Using a concept map to evaluate exploratory search
Proceedings of the third symposium on Information interaction in context
Biomedical information retrieval: the BioTracer approach
ITBAM'10 Proceedings of the First international conference on Information technology in bio- and medical informatics
Applying web usage mining for adaptive intranet navigation
IRFC'11 Proceedings of the Second international conference on Multidisciplinary information retrieval facility
Supporting biomedical information retrieval: the bioTracer approach
Transactions on large-scale data- and knowledge-centered systems IV
Looking for genre: the use of structural features during search tasks with Wikipedia
Proceedings of the 4th Information Interaction in Context Symposium
Technology Acceptance Model and Determinants of Technology Rejection
International Journal of Information Systems and Social Change
Human-Computer interaction view on information retrieval evaluation
PROMISE'12 Proceedings of the 2012 international conference on Information Retrieval Meets Information Visualization
Examining the generalizability of the User Engagement Scale (UES) in exploratory search
Information Processing and Management: an International Journal
You have e-mail, what happens next? Tracking the eyes for genre
Information Processing and Management: an International Journal
Hi-index | 0.00 |
The questionnaire is an important technique for gathering data from subjects during interactive information retrieval (IR) experiments. Research in survey methodology, public opinion polling and psychology has demonstrated a number of response biases and behaviors that subjects exhibit when responding to questionnaires. Furthermore, research in human-computer interaction has demonstrated that subjects tend to inflate their ratings of systems when completing usability questionnaires. In this study we investigate the relationship between questionnaire mode and subjects' responses to a usability questionnaire comprised of closed and open questions administered during an interactive IR experiment. Three questionnaire modes (pen-and-paper, electronic and interview) were explored with 51 subjects who used one of two information retrieval systems. Results showed that subjects' quantitative evaluations of systems were significantly lower in the interview mode than in the electronic mode. With respect to open questions, subjects in the interview mode used significantly more words than subjects in the pen-and-paper or electronic modes to communicate their responses, and communicated a significantly higher number of response units, even though the total number of unique response units was roughly the same across condition. Finally, results showed that subjects in the pen-and-paper mode were the most efficient in communicating their responses to open questions. These results suggest that researchers should use the interview mode to elicit responses to closed questions from subjects and either pen-and-paper or electronic modes to elicit responses to open questions.