Utility assessment in TRANSTAC: using a set of complementary methods

Authors:
Michelle Potts Steves;Emile Morse
Affiliations:
National Institute of Standards and Technology, Gaithersburg, Maryland;National Institute of Standards and Technology, Gaithersburg, Maryland
Venue:
PerMIS '09 Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems
Year:
2009

Citing 5
Cited 2

Semi-structured interviewing for user-centered design

interactions
A Practical Guide to Usability Testing

A Practical Guide to Usability Testing
Metrics for evaluating human information interaction systems

Interacting with Computers
Applying SCORE to field-based performance evaluations of soldier worn sensor technologies: Field Reports

Journal of Field Robotics
Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics

Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics

Lessons learned in evaluating DARPA advanced military technologies

Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop
Evaluation methodology and metrics employed to assess the TRANSTAC two-way, speech-to-speech translation systems

Computer Speech and Language

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper describes the methods used during formative utility assessments for speech-to-speech, real-time translation systems intended for tactical use. The systems, while still prototypical, had hardware platforms that could be exercised indoors or outdoors by study participants with pertinent backgrounds and skills that are similar to the intended target users. English-speaking and foreign language subjects participated in exercises using the systems exchanging information without the aid of a human interpreter. Feedback on subjects' experiences during the dialog interactions was collected via two primary methods: survey questionnaires and semi-structured interviews. In this paper, we describe our human-centered approach to utility assessment, how we combined the use of two feedback gathering methods, and discuss findings from use of this approach.