The evaluator effect in usability tests
CHI 98 Cconference Summary on Human Factors in Computing Systems
Usability Engineering
A Practical Guide to Usability Testing
A Practical Guide to Usability Testing
Usability Testing Essentials: Ready, Set...Test!
Usability Testing Essentials: Ready, Set...Test!
| Hi-index | 0.00 |
This paper documents a consistent and considerable evaluator effect in the ratings of usability problem severity carried out by experienced usability professionals. The CUE-9 study, conducted in 2011, showed a reasonable level of rater agreement in problem identification, but the severity assigned to problems varied wildly from "not a problem" to "disaster". This paper documents the variations we observed and the need for a better rating scale and better training. The paper calls for more caution when using extreme ratings. Our results also show that simple attempts to fix the traditional rating scales may not work.