Challenges in Evaluating Complex Decision Support Systems: Lessons from Design-a-Trial

  • Authors:
  • Henry W. W. Potts;Jeremy C. Wyatt;Douglas G. Altman

  • Affiliations:
  • -;-;-

  • Venue:
  • AIME '01 Proceedings of the 8th Conference on AI in Medicine in Europe: Artificial Intelligence Medicine
  • Year:
  • 2001

Quantified Score

Hi-index 0.00

Visualization

Abstract

Decision support system developers and users agree on the need for rigorous evaluations of performance and impact. Evaluating simple reminder systems is relatively easy because there is usually a gold standard of decision quality. However, when a system generates complex output (such as a critique or graphical report), it is much less obvious how to evaluate it. We discuss some generic problems and how one might resolve them, using as a case study Design-a-Trial, a DSS to help clinicians write a lengthy trial protocol.