Methodology for evaluating static six-degree-of-freedom (6DoF) perception systems

Authors:
Tommy Chang;Tsai Hong;Joe Falco;Michael Shneier;Mili Shah;Roger Eastman
Affiliations:
National Institute of Standard and Technology, Gaithersburg, Maryland;National Institute of Standard and Technology, Gaithersburg, Maryland;National Institute of Standard and Technology, Gaithersburg, Maryland;National Institute of Standard and Technology, Gaithersburg, Maryland;Loyola University Maryland, Baltimore, Maryland;Loyola University Maryland, Baltimore, Maryland
Venue:
Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop
Year:
2010

Citing 11
Cited 1

A comparative study of the robustness of two pose estimation techniques

Machine Vision and Applications - Special issue on performance evaluation
Model-Based Stereo-Tracking of Non-Polyhedral Objects for Automatic Disassembly Experiments

International Journal of Computer Vision - Special issue on image-based servoing
The Distribution of Target Registration Error in Rigid-Body, Point-Based Registration

IPMI '99 Proceedings of the 16th International Conference on Information Processing in Medical Imaging
Learning the Probability of Correspondences without Ground Truth

ICCV '05 Proceedings of the Tenth IEEE International Conference on Computer Vision - Volume 2
A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms

CVPR '06 Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1
Performance characterization in computer vision: A guide to best practices

Computer Vision and Image Understanding
Comparison of local image descriptors for full 6 degree-of-freedom pose estimation

ICRA'09 Proceedings of the 2009 IEEE international conference on Robotics and Automation
Dynamic 6DOF metrology for evaluating a visual servoing system

PerMIS '08 Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems
Mathematical metrology for evaluating a 6DOF visual servoing system

PerMIS '09 Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems
A framework for performance evaluation of model-based optical trackers

EGVE'08 Proceedings of the 14th Eurographics conference on Virtual Environments
An experimental comparison of three optical trackers for model based pose determination in virtual reality

EGVE'04 Proceedings of the Tenth Eurographics conference on Virtual Environments

Ground truth for evaluating 6 degrees of freedom pose estimation systems

Proceedings of the Workshop on Performance Metrics for Intelligent Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper, we apply two fundamental approaches toward evaluating a static, vision based, six-degree-of-freedom (6DoF) pose determination system that measures the position and orientation of a part. The first approach uses groundtruth carefully obtained from a laser tracker and the second approach doesn't use any external groundtruth. The evaluation procedure focuses on characterizing both the system's accuracy and precision as well as the effect of object viewpoints. For the groundtruth method, we first use a laser tracker for system calibration and then compare the calibrated output with the surveyed pose. In the method without external groundtruth, we evaluate the effect of viewpoint factors on the system's performance.