Sensing techniques for mobile interaction
UIST '00 Proceedings of the 13th annual ACM symposium on User interface software and technology
Extracting usability information from user interface events
ACM Computing Surveys (CSUR)
WebQuilt: A proxy-based approach to remote web usability testing
ACM Transactions on Information Systems (TOIS)
The state of the art in automating usability evaluation of user interfaces
ACM Computing Surveys (CSUR)
Evolution of web site design patterns
ACM Transactions on Information Systems (TOIS)
Proceedings of the 15th international conference on World Wide Web
Crowdsourcing user studies with Mechanical Turk
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A Systematic Review of Usability Evaluation in Web Development
WISE '08 Proceedings of the 2008 international workshops on Web Information Systems Engineering
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
TurKit: human computation algorithms on mechanical turk
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Web usability probe: a tool for supporting remote usability evaluation of web sites
INTERACT'11 Proceedings of the 13th IFIP TC 13 international conference on Human-computer interaction - Volume Part IV
Collaboratively crowdsourcing workflows with turkomatic
Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
Proceedings of the 4th ACM SIGCHI symposium on Engineering interactive computing systems
Crowdsourced web site evaluation with crowdstudy
ICWE'12 Proceedings of the 12th international conference on Web Engineering
AutoMan: a platform for integrating human-based and digital computation
Proceedings of the ACM international conference on Object oriented programming systems languages and applications
Crowdsourcing performance evaluations of user interfaces
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems
W3touch: metrics-based web page adaptation for touch
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
| Hi-index | 0.00 |
While traditional usability testing methods can be both time consuming and expensive, tools for automated usability evaluation tend to oversimplify the problem by limiting themselves to supporting only certain evaluation criteria, settings, tasks and scenarios. We present CrowdStudy, a general web toolkit that combines support for automated usability testing with crowdsourcing to facilitate large-scale online user testing. CrowdStudy is based on existing crowdsourcing techniques for recruiting workers and guiding them through complex tasks, but implements mechanisms specifically designed for usability studies, allowing testers to control user sampling and conduct evaluations for particular contexts of use. Our toolkit provides support for context-aware data collection and analysis based on an extensible set of metrics, as well as tools for managing, reviewing and analysing any collected data. The paper demonstrates several useful features of CrowdStudy for two different scenarios, and discusses the benefits and tradeoffs of using crowdsourced evaluation.