PerMIS '08 Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems
Ontological perspectives for autonomy performance
PerMIS '08 Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems
PerMIS '08 Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems
Evaluating the RoboCup 2009 Virtual Robot Rescue Competition
PerMIS '09 Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems
Evaluating speech translation systems: applying SCORE to TRANSTAC technologies
PerMIS '09 Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems
PerMIS '09 Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems
Utility assessment in TRANSTAC: using a set of complementary methods
PerMIS '09 Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems
Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop
Computer Speech and Language
| Hi-index | 0.00 |
Soldiers are often asked to perform missions that last many hours and are extremely stressful. After a mission is complete, the soldiers are typically asked to provide a report describing the most important things that happened during the mission. Due to the various stresses associated with military missions, there are undoubtedly many instances in which important information is missed or not reported and, therefore, not available for use when planning future missions. The ASSIST (Advanced Soldier Sensor Information System and Sensors Technology) program is addressing this challenge by instrumenting soldiers with sensors that they can wear directly on their uniforms. During the mission, the sensors continuously record what is going on around the soldier. With this information, soldiers are able to give more accurate reports without relying solely on their memory. In order for systems like this (often termed autonomous or intelligent systems) to be successful, they must be comprehensively and quantitatively evaluated to ensure that they will function appropriately and as expected in a wartime environment. The primary contribution of this paper is to introduce and define a framework and approach to performance evaluation called SCORE (System, Component, and Operationally Relevant Evaluation) and describe the results of applying it to evaluate the ASSIST technology. As the name implies, SCORE is built around the premise that, in order to get a true picture of how a system performs in the field, it must be evaluated at the component level, the system level, and in operationally relevant environments. The SCORE framework provides proven techniques to aid in the performance evaluation of many types of intelligent systems. To date, SCORE has only been applied to technologies under development (formative evaluation), but the authors believe that this approach would lend itself equally well to the evaluation of technologies ready to be fielded (summative evaluation). © 2007 Wiley Periodicals, Inc.