Remote evaluation: the network as an extension of the usability laboratory
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Adaptation of Traditional Usability Testing Methods for Remote Testing
HICSS '01 Proceedings of the 34th Annual Hawaii International Conference on System Sciences ( HICSS-34)-Volume 5 - Volume 5
Using terminological feedback for web search refinement: a log-based study
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
WiIRE: the web interactive information retrieval experimentation system prototype
Information Processing and Management: an International Journal
TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing)
What happened to remote usability testing?: an empirical study of three methods
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Crowdsourcing user studies with Mechanical Turk
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
On the history of evaluation in IR
Journal of Information Science
How does clickthrough data reflect retrieval quality?
Proceedings of the 17th ACM conference on Information and knowledge management
Crowdsourcing for relevance evaluation
ACM SIGIR Forum
Let your users do the testing: a comparison of three remote asynchronous usability testing methods
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A comparison of query and term suggestion features for interactive searching
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Towards methods for the collective gathering and quality control of relevance assessments
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Relevance criteria for e-commerce: a crowdsourcing-based experimental analysis
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Methods for Evaluating Interactive Information Retrieval Systems with Users
Foundations and Trends in Information Retrieval
Are your participants gaming the system?: screening mechanical turk workers
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Who are the crowdworkers?: shifting demographics in mechanical turk
CHI '10 Extended Abstracts on Human Factors in Computing Systems
ViewSer: a tool for large-scale remote studies of web search result examination
CHI '11 Extended Abstracts on Human Factors in Computing Systems
ViewSer: enabling large-scale remote user studies of web search examination and interaction
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
The impact of distraction in natural environments on user experience research
TPDL'11 Proceedings of the 15th international conference on Theory and practice of digital libraries: research and advanced technology for digital libraries
| Hi-index | 0.01 |
We compare two delivery modes for interactive search system (ISS) experiments: remote and laboratory. Our study was completed by two groups of subjects from the same population. The first group completed the study remotely and the second group completed the study in the laboratory. We compare differences in participants, participation behaviors, search behaviors and evaluation behaviors. Overall, for most measures no significant differences were found, but there were some notable differences. Greater variance was observed in time taken and number of documents opened and saved by remote subjects. Lab subjects provided more favorable responses to exit questionnaire items and reported significantly higher satisfaction. Lab subjects also provided significantly longer responses to open questions, while remote subjects provided more null responses. These results suggest that many behaviors do not change significantly according to study mode and that results from remote ISS experiments are similar to those from laboratory experiments.