A New Method to Evaluate Human-Robot System Performance
Autonomous Robots
How to trust robots further than we can throw them
CHI '04 Extended Abstracts on Human Factors in Computing Systems
Human-Robot Teaming for Search and Rescue
IEEE Pervasive Computing
Common metrics for human-robot interaction
Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction
Whose job is it anyway? a study of human-robot interaction in a collaborative task
Human-Computer Interaction
Autonomy levels for unmanned systems (ALFUS) framework: safety and application issues
PerMIS '07 Proceedings of the 2007 Workshop on Performance Metrics for Intelligent Systems
Developments and Challenges for Autonomous Unmanned Vehicles: A Compendium
Developments and Challenges for Autonomous Unmanned Vehicles: A Compendium
Measures for UGV to UGV collaboration
Proceedings of the Workshop on Performance Metrics for Intelligent Systems
| Hi-index | 0.00 |
The robotics community benefits from common test methods and metrics of performance to focus their research. As a result, many performance tests, competitions, demonstrations, and analyses have been devised to measure the autonomy, intelligence, and overall effectiveness of robots. These range from robot soccer (football) to measuring the performance of a robot in computer simulations. However, many resultant designs are narrowly focused oroptimized against the specific tasks under consideration. In the Multi-Autonomous Ground-robotic International Challenge (MAGIC) 2010, the need to transition the technology beyond the laboratory and into contexts for which it had not specifically been designed or tested meant that a performance evaluation scheme was needed that avoided domain-specific tests. However, the scheme still had to retain the capacity to deliver an impartial, consistent, objective, and evidence-based assessment that rewarded individual and multivehicle autonomy. It was also important to maximize the understanding and outcomes for technologists, sponsors, and potential usersgained through after-action review. The need for real-time, simultaneous, and continuous tracking of multiple interacting entities in an urban environment and over 250,000 square meters in real time compounded the complexity of the task. This paper describes the scheme used to progressively down-select and finally rank the teams competing in this complex and “operationally realistic” challenge. © 2012 Wiley Periodicals, Inc. © 2012 Wiley Periodicals, Inc.