Evaluation criteria for human-automation performance metrics

  • Authors:
  • Birsen Donmez;Patricia E. Pina;M. L. Cummings

  • Affiliations:
  • MIT, Cambridge, MA;MIT, Cambridge, MA;MIT, Cambridge, MA

  • Venue:
  • PerMIS '08 Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

Previous research has identified broad metric classes for human-automation performance to facilitate metric selection, as well as understanding and comparison of research results. However, there is still lack of an objective method for selecting the most efficient set of metrics. This research identifies and presents a list of evaluation criteria that can help determine the quality of a metric in terms of experimental constraints, comprehensive understanding, construct validity, statistical efficiency, and measurement technique efficiency. Future research will build on these evaluation criteria and existing generic metric classes to develop a cost-benefit analysis approach that can be used for metric selection.