Controlled experiment replication in evaluation of e-learning system's educational influence

Authors:
Ani Grubišić;Slavomir Stankov;Marko Rosić;Branko itko
Affiliations:
Faculty of Natural Science, Mathematics and Kinesiology, University of Split, Teslina 12, 21000 Split, Croatia;Faculty of Natural Science, Mathematics and Kinesiology, University of Split, Teslina 12, 21000 Split, Croatia;Faculty of Natural Science, Mathematics and Kinesiology, University of Split, Teslina 12, 21000 Split, Croatia;Faculty of Natural Science, Mathematics and Kinesiology, University of Split, Teslina 12, 21000 Split, Croatia
Venue:
Computers & Education
Year:
2009

Citing 8
Cited 2

Some principles of intelligent tutoring

Artificial intelligence and education; vol. 1: learning environments and tutoring systems
Experimental design and analysis in software engineering: Part 2: how to set up and experiment

ACM SIGSOFT Software Engineering Notes
Designing Process Replication and Activation: A Quantitative Approach

IEEE Transactions on Software Engineering
A Review of Experimental Investigations into Object-Oriented Technology

Empirical Software Engineering
A Classification of Evaluation Methods for Intelligent Tutoring Systems

Software-Ergonomie '99, Design von Informationswelten, Gemeinsame Fachtagung des German Chapter of the ACM, der Gesellschaft für Informatik (GI) und der SAP AG
Having It All, Maybe: Design Tradeoffs in ITS Authoring Tools

ITS '96 Proceedings of the Third International Conference on Intelligent Tutoring Systems
An approach to automatic evaluation of educational influence

DIWED'06 Proceedings of the 6th WSEAS International Conference on Distance Learning and Web Engineering
e-Learning in Project Management Using Simulation Models: A Case Study Based on the Replication of an Experiment

IEEE Transactions on Education

A web-based learning tool improves student performance in statistics: A randomized masked trial

Computers & Education
Evaluating the integration of fuzzy logic into the student model of a web-based learning environment

Expert Systems with Applications: An International Journal

Quantified Score

Hi-index	0.00

Visualization

Abstract

We believe that every effectiveness evaluation should be replicated at least in order to verify the original results and to indicate evaluated e-learning system's advantages or disadvantages. This paper presents the methodology for conducting controlled experiment replication, as well as, results of a controlled experiment and an internal replication that investigated the effectiveness of intelligent authoring shell eXtended Tutor-Expert System (xTEx-Sys). The initial and the replicated experiment were based on our approach that combines classical two-group experimental design and with factoral design. A trait that distinguishes this approach from others is the existence of arbitrary number of checkpoint-tests to determine the effectiveness in intermediate states. We call it a pre-and-post test control group experimental design with checkpoint-tests. The gained results revealed small or even negative effect sizes, which could be explained by the fact that the xTEx-Sys's domain knowledge presentation is rather novel for students and therefore difficult to grasp and apply in earlier phases of the experiment. In order to develop and improve the xTEx-Sys, further experiments must be conducted.