OHSUMED: an interactive retrieval evaluation and new large test collection for research
SIGIR '94 Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval
Evaluation of evaluation in information retrieval
SIGIR '95 Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval
Image organization and retrieval with automatically constructed feature vectors
SIGIR '96 Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval
Modeling and retrieving images by content
Information Processing and Management: an International Journal
Corpus-based stemming using cooccurrence of word variants
ACM Transactions on Information Systems (TOIS)
Comparison of edge detectors: a methodology and initial study
Computer Vision and Image Understanding
Content-based image retrieval using a composite color-shape approach
Information Processing and Management: an International Journal
Attributes of images in describing tasks
Information Processing and Management: an International Journal
Spatial querying for image retrieval: a user-oriented evaluation
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Fuzzy content-based retrieval in image databases
Information Processing and Management: an International Journal
A relevance feedback mechanism for content-based image retrieval
Information Processing and Management: an International Journal
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Does organisation by similarity assist image browsing?
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Information Processing and Management: an International Journal - Special issue on interactivity at the text retrieval conference (TREC)
Information Processing and Management: an International Journal - Special issue on interactivity at the text retrieval conference (TREC)
Image searching on the Excite web search engine
Information Processing and Management: an International Journal
Reflections on Mira: interactive evaluation in information retrieval
Journal of the American Society for Information Science
Evaluating a content based image retrieval system
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
A conceptual framework and empirical research for classifying visual descriptors
Journal of the American Society for Information Science and Technology
Modern Information Retrieval
End-User Searching Challenges Indexing Practices inthe Digital Newspaper Photo Archive
Information Retrieval
User's relevance criteria in image retrieval in American history
Information Processing and Management: an International Journal - Issues of context in information retrieval
Information Processing and Management: an International Journal
CIVR '02 Proceedings of the International Conference on Image and Video Retrieval
The Truth about Corel - Evaluation in Image Retrieval
CIVR '02 Proceedings of the International Conference on Image and Video Retrieval
Modelling subjectivity in visual perception of orientation for image retrieval
Information Processing and Management: an International Journal - Modelling vagueness and subjectivity in information access
Search strategies in content-based image retrieval
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Image classification using hybrid neural networks
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach
IEEE Transactions on Pattern Analysis and Machine Intelligence
The Journal of Machine Learning Research
Content-based image classification using a neural network
Pattern Recognition Letters
IEEE Transactions on Image Processing
Image classification for content-based indexing
IEEE Transactions on Image Processing
CLAIRE: A modular support vector image indexing and classification system
ACM Transactions on Information Systems (TOIS)
An image-centred "search and indexation system" based in user's data and perceived emotion
HCC '08 Proceedings of the 3rd ACM international workshop on Human-centered computing
An attentive self-organizing neural model for text mining
Expert Systems with Applications: An International Journal
Hi-index | 0.00 |
In image retrieval, most systems lack user-centred evaluation since they are assessed by some chosen ground truth dataset. The results reported through precision and recall assessed against the ground truth are thought of as being an acceptable surrogate for the judgment of real users. Much current research focuses on automatically assigning keywords to images for enhancing retrieval effectiveness. However, evaluation methods are usually based on system-level assessment, e.g. classification accuracy based on some chosen ground truth dataset. In this paper, we present a qualitative evaluation methodology for automatic image indexing systems. The automatic indexing task is formulated as one of image annotation, or automatic metadata generation for images. The evaluation is composed of two individual methods. First, the automatic indexing annotation results are assessed by human subjects. Second, the subjects are asked to annotate some chosen images as the test set whose annotations are used as ground truth. Then, the system is tested by the test set whose annotation results are judged against the ground truth. Only one of these methods is reported for most systems on which user-centred evaluation are conducted. We believe that both methods need to be considered for full evaluation. We also provide an example evaluation of our system based on this methodology. According to this study, our proposed evaluation methodology is able to provide deeper understanding of the system's performance.