Evolution of the SCORE framework to enhance field-based performance evaluations of emerging technologies

Authors:
Brian A. Weiss;Craig Schlenoff
Affiliations:
National Institute of Standards and Technology, Gaithersburg, Maryland;National Institute of Standards and Technology, Gaithersburg, Maryland
Venue:
PerMIS '08 Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems
Year:
2008

Citing 2
Cited 6

A Framework for Evaluating Collaborative Systems in the Real World

HICSS '05 Proceedings of the Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS'05) - Track 1 - Volume 01
Applying SCORE to field-based performance evaluations of soldier worn sensor technologies: Field Reports

Journal of Field Robotics

Ontological perspectives for autonomy performance

PerMIS '08 Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems
Performance measures framework for unmanned systems (PerMFUS): initial perspective

PerMIS '09 Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems
Evaluating speech translation systems: applying SCORE to TRANSTAC technologies

PerMIS '09 Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems
The impact of evaluation scenario development on the quantitative performance of speech translation systems prescribed by the SCORE framework

PerMIS '09 Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems
The multi-relationship evaluation design framework: creating evaluation blueprints to assess advanced and intelligent technologies

Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop
Evaluation methodology and metrics employed to assess the TRANSTAC two-way, speech-to-speech translation systems

Computer Speech and Language

Quantified Score

Hi-index	0.00

Visualization

Abstract

NIST has developed the System, Component, and Operationally-Relevant Evaluations (SCORE) framework as a formal guide for designing evaluations of emerging technologies. SCORE captures both technical performance and end-user utility assessments of systems and their components within controlled and realistic environments. Its purpose is to present an extensive (but not necessarily exhaustive) picture of how a system would behave in a realistic operating environment. The framework has been applied to numerous evaluation efforts over the past three years producing valuable quantitative and qualitative metrics. This paper will present the building blocks of the SCORE methodology including the system goals and design criteria that drive the evaluation design process. An evolution of the SCORE framework in capturing utility assessments at the capability level of a system will also be presented. Examples will be shown of SCORE's successful application to the evaluation of the soldier-worn sensor systems and two-way, free-form spoken language translation technologies.