Object Recognition from Local Scale-Invariant Features
ICCV '99 Proceedings of the International Conference on Computer Vision-Volume 2 - Volume 2
Incremental Algorithms for Hierarchical Classification
The Journal of Machine Learning Research
Introduction to Information Retrieval
Introduction to Information Retrieval
The MIR flickr retrieval evaluation
MIR '08 Proceedings of the 1st ACM international conference on Multimedia information retrieval
Overview of the CLEF 2009 large-scale visual concept detection and annotation task
CLEF'09 Proceedings of the 10th international conference on Cross-language evaluation forum: multimedia experiments
The University of Aamsterdam's concept detection system at ImageCLEF 2009
CLEF'09 Proceedings of the 10th international conference on Cross-language evaluation forum: multimedia experiments
UAIC at ImageCLEF 2009 photo annotation task
CLEF'09 Proceedings of the 10th international conference on Cross-language evaluation forum: multimedia experiments
Mining Multilevel Image Semantics via Hierarchical Classification
IEEE Transactions on Multimedia
New trends and ideas in visual concept detection: the MIR flickr retrieval evaluation initiative
Proceedings of the international conference on Multimedia information retrieval
The effect of semantic relatedness measures on multi-label classification evaluation
Proceedings of the ACM International Conference on Image and Video Retrieval
Overview of the CLEF 2009 large-scale visual concept detection and annotation task
CLEF'09 Proceedings of the 10th international conference on Cross-language evaluation forum: multimedia experiments
Overview of the photo annotation task in imageCLEF@ICPR
ICPR'10 Proceedings of the 20th International conference on Recognizing patterns in signals, speech, images, and videos
Graph-based methods for the automatic annotation and retrieval of art prints
Proceedings of the 1st ACM International Conference on Multimedia Retrieval
On the stratification of multi-label data
ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part III
Making image to class distance comparable
ICONIP'11 Proceedings of the 18th international conference on Neural Information Processing - Volume Part II
Artistic image classification: an analysis on the PRINTART database
ECCV'12 Proceedings of the 12th European conference on Computer Vision - Volume Part IV
| Hi-index | 0.00 |
With the steadily increasing amount of multimedia documents on the web and at home, the need for reliable semantic indexing methods that assign multiple keywords to a document grows. The performance of existing approaches is often measured with standard evaluation measures of the information retrieval community. In a case study on image annotation, we show the behaviour of 13 different evaluation measures and point out their strengths and weaknesses. For the analysis, data from 19 research groups that participated in the ImageCLEF Photo Annotation Task are utilized together with several configurations based on random numbers. A recently proposed ontology-based measure was investigated that incorporates structure information, relationships from the ontology and the agreement between annotators for a concept and compared to a hierarchical variant. The results for the hierarchical measure are not competitive. The ontology-based results assign good scores to the systems that got also good ranks in the other measures like the example-based F-measure. For concept-based evaluation, stable results could be obtained for MAP concerning random numbers and the number of annotated labels. The AUC measure shows good evaluation characteristics in case all annotations contain confidence values.