Using Evaluation to Shape ITS Design: Results and Experiences with SQL-Tutor

  • Authors:

  • Antonija Mitrovic;Brent Martin;Michael Mayo

  • Affiliations:

  • Intelligent Computer Tutoring Group, Department of Computer Science, University of Canterbury, Private Bag 4800, Christchurch, New Zealand. E-mail: tanja@cosc.canterbury.ac.nz;Intelligent Computer Tutoring Group, Department of Computer Science, University of Canterbury, Private Bag 4800, Christchurch, New Zealand. E-mail: bim20@cosc.canterbury.ac.nz;Intelligent Computer Tutoring Group, Department of Computer Science, University of Canterbury, Private Bag 4800, Christchurch, New Zealand. E-mail: mmayo@cosc.canterbury.ac.nz

  • Venue:
  • User Modeling and User-Adapted Interaction
  • Year:
  • 2002

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper presents the results of three evaluation studies performed during 1998 and 1999 on SQL-Tutor, an intelligent tutoring system for the SQL database language. We have evaluated the system in the context of genuine courses, and used the results to further refine the system. The main goal of our research has been the exploration and extension of Constraint-Based Modeling (CBM), a student modeling approach proposed by Ohlsson (1994). SQL-Tutor provided us with experiences of using CBM, and we used it to extend the approach in several important ways. The main goal of all three evaluation studies was to determine how well CBM supported student learning. We have obtained positive results. The students who learnt with SQL-Tutor in the first study performed significantly better than those who did not when assessed by a subsequent classroom examination. Furthermore, the analysis of students' learning shows that CBM has a sound psychological foundation.Besides the evaluation of CBM, we also evaluated the improvements in terms of student assessments of the usefulness of the system and evaluated various techniques used in SQL-Tutor. In the second study, we evaluated the effectiveness of feedback provided to the students. This study showed that high-level advice is most beneficial to students' learning. The focus of the third study was different. We extended CBM to support long-term modeling of student knowledge, and used this extension to develop an adaptive problem-selection strategy. The study revealed the benefits of this strategy in comparison with a simple heuristic strategy. We also reflect on our experiences in evaluating SQL-Tutor.