Metric selection for evaluating human supervisory control of unmanned vehicles

• Authors: Birsen Donmez;M. L. Cummings
• Venue: Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop
• Year: 2010

Broad metric classes were proposed in the literature in order to facilitate metric selection for evaluating human-autonomous vehicle interaction. However, there still lacks a systematic method for selecting an efficient set of metrics from the many metrics available. We previously identified a list of evaluation criteria that can help determine the quality of a metric, and generated a list of potential metric costs and benefits. Depending on research objectives and limitations, these costs and benefits can have different weights of importance. Through an experiment with subject matter experts, we investigated which metric characteristics human factors practitioners consider to be important in evaluating human supervisory control of unmanned vehicles. We also tested two different multi-criteria decision making methods to help practitioners assign subjective weights to the cost/benefit criteria. The majority of participants rated the evaluation criteria used in both tools as very useful. However, the majority of participants' metric selections before using the methods were the same as the suggestions provided by the methods. Since determining weights of metric importance is an inherently subjective process, even with objective computational tools, the real value of using such a tool may be reminding human factors practitioners of the important experimental criteria and relationships between these criteria that should be considered when designing an experiment.