Reliability of severity estimates for usability problems found by heuristic evaluation

Authors:
Jakob Nielsen
Affiliations:
Bellcore, Morristown, NJ
Venue:
CHI '92 Posters and Short Talks of the 1992 SIGCHI Conference on Human Factors in Computing Systems
Year:
1992

Citing 2
Cited 3

Heuristic evaluation of user interfaces

CHI '90 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Finding usability problems through heuristic evaluation

CHI '92 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared

CHI '93 Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems
Critic-proofing: how using critic reviews and game genres can refine heuristic evaluations

Futureplay '10 Proceedings of the International Academic Conference on the Future of Game Design and Technology
Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0

ACM Transactions on Accessible Computing (TACCESS)

Quantified Score

Hi-index	0.00

Visualization

Abstract

Ratings from single evaluators are very unreliable when usability specialists judge the severity of usability problems found by heuristic evaluation, but the mean severity rating from four evaluators gets within half a rating point of the true severity 95% of the time. Also, the evaluators do agree that usability problems found by heuristic evaluation are all real problems even though each rater had originally only identified a small proportion of the problems.