Measuring the effectiveness of error messages designed for novice programmers

  • Authors:

  • Guillaume Marceau;Kathi Fisler;Shriram Krishnamurthi

  • Affiliations:

  • Worcester Polytechnic Institute, Worcester, MA, USA;Worcester Polytechnic Institute, Worcester, MA, USA;Brown University, Providence, RI, USA

  • Venue:
  • Proceedings of the 42nd ACM technical symposium on Computer science education
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Good error messages are critical for novice programmers. Re-cognizing this, the DrRacket programming environment provides a series of pedagogically-inspired language subsets with error messages customized to each subset. We apply human-factors research methods to explore the effectiveness of these messages. Unlike existing work in this area, we study messages at a fine-grained level by analyzing the edits students make in response to various classes of errors. We present a rubric (which is not language specific) to evaluate student responses, apply it to a course-worth of student lab work, and describe what we have learned about using the rubric effectively. We also discuss some concrete observations on the effectiveness of these messages.