Refining the test phase of usability evaluation: how many subjects is enough?
Human Factors - Special issue: measurement in human factors
A mathematical model of the finding of usability problems
INTERCHI '93 Proceedings of the INTERCHI '93 conference on Human factors in computing systems
The evaluator effect in usability tests
CHI 98 Cconference Summary on Human Factors in Computing Systems
Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests
Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests
A Practical Guide to Usability Testing
A Practical Guide to Usability Testing
Comparative evaluation of usability tests
CHI '99 Extended Abstracts on Human Factors in Computing Systems
On the reliability of usability testing
CHI '01 Extended Abstracts on Human Factors in Computing Systems
Testing web sites: five users is nowhere near enough
CHI '01 Extended Abstracts on Human Factors in Computing Systems
Damaged merchandise? a review of experiments that compare usability evaluation methods
Human-Computer Interaction
Describing usability problems: are we sending the right message?
interactions - All systems go: how Wall street will benefit from user-centered design
Making a difference: a survey of the usability profession in Sweden
Proceedings of the third Nordic conference on Human-computer interaction
Analysis of strategies for improving and estimating the effectiveness of heuristic evaluation
Proceedings of the third Nordic conference on Human-computer interaction
Comparing usability problems and redesign proposals as input to practical systems development
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A method to standardize usability metrics into a single score
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Automated summative usability studies: an empirical evaluation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Tips and tricks for better usability test recommendations
CHI '06 Extended Abstracts on Human Factors in Computing Systems
What do usability evaluators do in practice?: an explorative study of think-aloud testing
DIS '06 Proceedings of the 6th conference on Designing Interactive systems
It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field
Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles
How HCI-practitioners want to evaluate their own practice
Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles
Heuristic evaluation: Comparing ways of finding and reporting usability problems
Interacting with Computers
Towards an empirical method of efficiency testing of system parts: A methodological study
Interacting with Computers
Usability testing: what have we overlooked?
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Recommendations on recommendations
CHI '07 Extended Abstracts on Human Factors in Computing Systems
A comparative evaluation of heuristic-based usability inspection methods
CHI '08 Extended Abstracts on Human Factors in Computing Systems
Comparative usability evaluation (CUE-4)
Behaviour & Information Technology
Comparison of techniques for matching of usability problem descriptions
Interacting with Computers
The theoretical foundation and validity of a component-based usability questionnaire
Behaviour & Information Technology
Supporting novice usability practitioners with usability engineering tools
International Journal of Human-Computer Studies
Is the `Figure of Merit' Really That Meritorious?
INTERACT '09 Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part I
Interaction walkthrough: evaluation of safety critical interactive systems
DSVIS'06 Proceedings of the 13th international conference on Interactive systems: Design, specification, and verification
Evaluating usability evaluation methods: criteria, method and a case study
HCI'07 Proceedings of the 12th international conference on Human-computer interaction: interaction design and usability
Model-driven adaptation for plastic user interfaces
INTERACT'07 Proceedings of the 11th IFIP TC 13 international conference on Human-computer interaction
Usability of an evidence-based practice website on a pediatric neuroscience unit
USAB'07 Proceedings of the 3rd Human-computer interaction and usability engineering of the Austrian computer society conference on HCI and usability for medicine and health care
Comparison of playtesting and expert review methods in mobile game evaluation
Proceedings of the 3rd International Conference on Fun and Games
Weak inter-rater reliability in heuristic evaluation of video games
CHI '11 Extended Abstracts on Human Factors in Computing Systems
Discourse Variations Between Usability Tests and Usability Reports
Journal of Usability Studies
Towards the maturation of IT usability evaluation (MAUSE)
INTERACT'05 Proceedings of the 2005 IFIP TC13 international conference on Human-Computer Interaction
Designing and Evaluating Mobile Interaction: Challenges and Trends
Foundations and Trends in Human-Computer Interaction
E-learning and market orientation in higher education
Education and Information Technologies
Journal of Biomedical Informatics
| Hi-index | 0.00 |
This paper reports on a study assessing the consistency of usability testing across organisations. Nine independent organisations evaluated the usability of the same website, Microsoft Hotmail. The results document a wide difference in selection and application of methodology, resources applied, and problems reported. The organizations reported 310 different usability problems. Only two problems were reported by six or more organizations, while 232 problems (75%) were uniquely reported, that is, no two teams reported the same problem. Some of the unique findings were classified as serious. Even the tasks used by most or all teams produced very different results - around 70% of the findings for each of these tasks were unique. Our main conclusion is that our simple assumption that we are all doing the same and getting the same results in a usability test is plainly wrong.