Parametrization of an image understanding quality metric with a subjective evaluation

Authors:
B. Hemery;H. Laurent;B. Emile;C. Rosenberger
Affiliations:
Université de Caen Basse-Normandie, UMR 6072 GREYC, F-14032 Caen, France and ENSICAEN, UMR 6072 GREYC, F-14050 Caen, France and CNRS, UMR 6072 GREYC, F-14032 Caen, France;PRISME Laboratory, ENSI de Bourges, Université d'Orléans, France;PRISME Laboratory, IUT de l'Indre, Université d'Orléans, France;Université de Caen Basse-Normandie, UMR 6072 GREYC, F-14032 Caen, France and ENSICAEN, UMR 6072 GREYC, F-14050 Caen, France and CNRS, UMR 6072 GREYC, F-14032 Caen, France
Venue:
Pattern Recognition Letters
Year:
2013

Citing 10
Cited 0

Distance measures for signal processing and pattern recognition

Signal Processing
Assessing agreement on classification tasks: the kappa statistic

Computational Linguistics
A New Metric for Grey-Scale Image Comparison

International Journal of Computer Vision
Histograms of Oriented Gradients for Human Detection

CVPR '05 Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1 - Volume 01
Object count/area graphs for the evaluation of object detection and segmentation algorithms

International Journal on Document Analysis and Recognition
A new supervised evaluation criterion for region based segmentation methods

ACIVS'07 Proceedings of the 9th international conference on Advanced concepts for intelligent vision systems
Evaluation metric for image understanding

ICIP'09 Proceedings of the 16th IEEE international conference on Image processing
Scale-invariant shape features for recognition of object categories

CVPR'04 Proceedings of the 2004 IEEE computer society conference on Computer vision and pattern recognition
Evaluation of the quality of ultrasound image compression by fusion of criteria with a genetic algorithm

ICAPR'05 Proceedings of the Third international conference on Pattern Recognition and Image Analysis - Volume Part II
Detecting moving objects, ghosts, and shadows in video streams

IEEE Transactions on Pattern Analysis and Machine Intelligence

Quantified Score

Hi-index	0.10

Visualization

Abstract

Image understanding has many real industrial applications (video-monitoring, image retrieval, etc.). Given an image and an associated ground truth, it is possible to quanjpgy the quality of understanding results provided by different algorithms or parameters. To this end, it is necessary to take into account many factors for each object in the image: localization and recognition errors and under or over-detection of objects. In order to define an evaluation metric for quanjpgying the quality of an image understanding result, we have to set, as for example, the weights of each kind of error in the global score. For a correct parameters setting of an evaluation metric we defined previously, we conducted a subjective evaluation of image understanding results involving many experts in image processing. We present in this paper the developed method and analyze the obtained results to weight the various errors in an appropriate way. We show the benefit of this kind of study to define the correct parameters of the metric in order to have a judgment as reliable the one provided by experts. Experimental results on many images from the PASCAL VOC Challenge show the good behavior of this metric compared to the human judgment.