The MIR flickr retrieval evaluation

Authors:
Mark J. Huiskes;Michael S. Lew
Affiliations:
Leiden University, Leiden, Netherlands;Leiden University, Leiden, Netherlands
Venue:
MIR '08 Proceedings of the 1st ACM international conference on Multimedia information retrieval
Year:
2008

Citing 8
Cited 99

Benchmarking for Content-Based Visual Information Search

VISUAL '00 Proceedings of the 4th International Conference on Advances in Visual Information Systems
The Truth about Corel - Evaluation in Image Retrieval

CIVR '02 Proceedings of the International Conference on Image and Video Retrieval
Content-based multimedia information retrieval: State of the art and challenges

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)
Benchmarking image and video retrieval: an overview

MIR '06 Proceedings of the 8th ACM international workshop on Multimedia information retrieval
Usage-oriented multimedia information retrieval technological evaluation

MIR '06 Proceedings of the 8th ACM international workshop on Multimedia information retrieval
Evaluation campaigns and TRECVid

MIR '06 Proceedings of the 8th ACM international workshop on Multimedia information retrieval
Performance evaluation of relevance feedback methods

CIVR '08 Proceedings of the 2008 international conference on Content-based image and video retrieval
Classification and annotation of digital photos using optical context data

CIVR '08 Proceedings of the 2008 international conference on Content-based image and video retrieval

An automatic translation of tags for multimedia contents using folksonomy networks

Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Classification and quality assessment of high quality digital photographs

MM '09 Proceedings of the 17th ACM international conference on Multimedia
GaZIR: gaze-based zooming interface for image retrieval

Proceedings of the 2009 international conference on Multimodal interfaces
Improving Folksonomies Using Formal Knowledge: A Case Study on Search

ASWC '09 Proceedings of the 4th Asian Conference on The Semantic Web
Using one-class SVM outliers detection for verification of collaboratively tagged image training sets

ICME'09 Proceedings of the 2009 IEEE international conference on Multimedia and Expo
Performance measures for multilabel evaluation: a case study in the area of image classification

Proceedings of the international conference on Multimedia information retrieval
Multimodal medical image retrieval: image categorization to improve search precision

Proceedings of the international conference on Multimedia information retrieval
QbS: searching for known images using user-drawn sketches

Proceedings of the international conference on Multimedia information retrieval
New trends and ideas in visual concept detection: the MIR flickr retrieval evaluation initiative

Proceedings of the international conference on Multimedia information retrieval
Image annotation with tagprop on the MIRFLICKR set

Proceedings of the international conference on Multimedia information retrieval
Automatically annotating the MIR Flickr dataset: experimental protocols, openly available data and semantic spaces

Proceedings of the international conference on Multimedia information retrieval
How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation

Proceedings of the international conference on Multimedia information retrieval
MAP-based image tag recommendation using a visual folksonomy

Pattern Recognition Letters
THESEUS meets ImageCLEF: combining evaluation strategies for a new visual concept detection task 2009

CLEF'08 Proceedings of the 9th Cross-language evaluation forum conference on Evaluating systems for multilingual and multimodal information access
Signature Quadratic Form Distance

Proceedings of the ACM International Conference on Image and Video Retrieval
Optimized block-based connected components labeling with decision trees

IEEE Transactions on Image Processing
Image and collateral text in support of auto-annotation and sentiment analysis

TextGraphs-5 Proceedings of the 2010 Workshop on Graph-based Methods for Natural Language Processing
Image retrieval at memory's edge: known image search based on user-drawn sketches

CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Novel local features with hybrid sampling technique for image retrieval

CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Image tag refinement towards low-rank, content-tag prior and error sparsity

Proceedings of the international conference on Multimedia
Quantifying tag representativeness of visual content of social images

Proceedings of the international conference on Multimedia
Assisted news reading with automated illustration

Proceedings of the international conference on Multimedia
Overview of the CLEF 2009 large-scale visual concept detection and annotation task

CLEF'09 Proceedings of the 10th international conference on Cross-language evaluation forum: multimedia experiments
Tag refinement in an image folksonomy using visual similarity and tag co-occurrence statistics

Image Communication
Automatic image semantic interpretation using social action and tagging data

Multimedia Tools and Applications
Overview of the photo annotation task in imageCLEF@ICPR

ICPR'10 Proceedings of the 20th International conference on Recognizing patterns in signals, speech, images, and videos
L2-signature quadratic form distance for efficient query processing in very large multimedia databases

MMM'11 Proceedings of the 17th international conference on Advances in multimedia modeling - Volume Part I
Efficient clustering and quantisation of SIFT features: exploiting characteristics of the SIFT descriptor and interest region detectors under image inversion

Proceedings of the 1st ACM International Conference on Multimedia Retrieval
Indexing the signature quadratic form distance for efficient content-based multimedia retrieval

Proceedings of the 1st ACM International Conference on Multimedia Retrieval
Active learning through notes data in Flickr: an effortless training data acquisition approach for object localization

Proceedings of the 1st ACM International Conference on Multimedia Retrieval
RetrievalLab: a programming tool for content based retrieval

Proceedings of the 1st ACM International Conference on Multimedia Retrieval
Ptolemaic indexing of the signature quadratic form distance

Proceedings of the Fourth International Conference on SImilarity Search and APplications
Parameterized earth mover's distance for efficient metric space indexing

Proceedings of the Fourth International Conference on SImilarity Search and APplications
Image annotation based on recommendation model

Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Growing hierarchical self-organizing map for images hierarchical clustering

ICCCI'11 Proceedings of the Third international conference on Computational collective intelligence: technologies and applications - Volume Part I
Evaluation platform for content-based image retrieval systems

TPDL'11 Proceedings of the 15th international conference on Theory and practice of digital libraries: research and advanced technology for digital libraries
Associating textual features with visual ones to improve affective image classification

ACII'11 Proceedings of the 4th international conference on Affective computing and intelligent interaction - Volume Part I
Retrieving and ranking unannotated images through collaboratively mining online search results

Proceedings of the 20th ACM international conference on Information and knowledge management
Multimodal representation, indexing, automated annotation and retrieval of image collections via non-negative matrix factorization

Neurocomputing
Context seeking with social tags

Proceedings of the fourth workshop on Exploiting semantic annotations in information retrieval
Purposive hidden-object-game: embedding human computation in popular game

MM '11 Proceedings of the 19th ACM international conference on Multimedia
Transfer tagging from image to video

MM '11 Proceedings of the 19th ACM international conference on Multimedia
Shared feature extraction for semi-supervised image classification

MM '11 Proceedings of the 19th ACM international conference on Multimedia
Shared feature extraction for semi-supervised image classification

MM '11 Proceedings of the 19th ACM international conference on Multimedia
PhotoFeel: feeling your photo collection with graph-based audiovisual flocking

MM '11 Proceedings of the 19th ACM international conference on Multimedia
Reading between the tags to predict real-world size-class for visually depicted objects in images

MM '11 Proceedings of the 19th ACM international conference on Multimedia
Learning to associate relevant photos to georeferenced textual documents

Proceedings of the 19th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems
Using manual and automated annotations to search images by semantic similarity

Multimedia Tools and Applications
Semantic concept detection for user-generated video content using a refined image folksonomy

MMM'10 Proceedings of the 16th international conference on Advances in Multimedia Modeling
What's with the free images?: a study of Flickr's creative commons attribution images

Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments
On stability of adaptive similarity measures for content-based image retrieval

MMM'12 Proceedings of the 18th international conference on Advances in Multimedia Modeling
Classifying high quality photographs by creative exposure themes

FDIA'09 Proceedings of the Third BCS-IRSG conference on Future Directions in Information Access
An interactive paper and digital pen interface for query-by-sketch image retrieval

ECIR'12 Proceedings of the 34th European conference on Advances in Information Retrieval
Image abstraction in crossmedia retrieval for text illustration

ECIR'12 Proceedings of the 34th European conference on Advances in Information Retrieval
A user interface for query-by-sketch based image retrieval with color sketches

ECIR'12 Proceedings of the 34th European conference on Advances in Information Retrieval
Multimodal feature generation framework for semantic image classification

Proceedings of the 2nd ACM International Conference on Multimedia Retrieval
Multimodal fusion for image retrieval using matrix factorization

Proceedings of the 2nd ACM International Conference on Multimedia Retrieval
Assistive tagging: A survey of multimedia tagging with human-computer joint exploration

ACM Computing Surveys (CSUR)
Nearest-neighbor method using multiple neighborhood similarities for social media data mining

Neurocomputing
Constructing visual tag dictionary by mining community-contributed media corpus

Neurocomputing
Combining CPU and GPU architectures for fast similarity search

Distributed and Parallel Databases
A universal technique to hide traces of histogram-based image manipulations

Proceedings of the on Multimedia and security
On the consistency and features of image similarity

Proceedings of the 4th Information Interaction in Context Symposium
Automatic annotation of tagged content using predefined semantic concepts

Proceedings of the 18th Brazilian symposium on Multimedia and the web
Joint statistical analysis of images and keywords with applications in semantic image enhancement

Proceedings of the 20th ACM international conference on Multimedia
Snap-and-ask: answering multimodal question by naming visual instance

Proceedings of the 20th ACM international conference on Multimedia
Interactive exploration of large remote image databases

Proceedings of the 20th ACM international conference on Multimedia
Semantic awareness for automatic image interpretation

Proceedings of the 20th ACM international conference on Multimedia
DC stream based JPEG compressed domain image retrieval

AMT'12 Proceedings of the 8th international conference on Active Media Technology
Performance Evaluation of Relevance Feedback for Image Retrieval by "Real-World" Multi-Tagged Image Datasets

International Journal of Multimedia Data Engineering & Management
Enhancing Folksonomy-Based Content Retrieval with Semantic Web Technology

International Journal on Semantic Web & Information Systems
Automatic clustering for digital photograph collections using time and content information

Proceedings of the 7th International Conference on Ubiquitous Information Management and Communication
Comparison of mid-level feature coding approaches and pooling strategies in visual concept detection

Computer Vision and Image Understanding
Multimodal recognition of visual concepts using histograms of textual concepts and selective weighted late fusion scheme

Computer Vision and Image Understanding
Editorial: Special issue on visual concept detection in the MIRFLICKR/ImageCLEF benchmark

Computer Vision and Image Understanding
Pooling in image representation: The visual codeword point of view

Computer Vision and Image Understanding
CONTENTUS--technologies for next generation multimedia libraries

Multimedia Tools and Applications
Multifeature analysis and semantic context learning for image classification

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)
Effective transfer tagging from image to video

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)
Tag completion based on belief theory and neighbor voting

Proceedings of the 3rd ACM conference on International conference on multimedia retrieval
The Mosaic Test: measuring the effectiveness of colour-based image retrieval

Multimedia Tools and Applications
A picture is worth a thousand tags: automatic web based image tag expansion

ACCV'12 Proceedings of the 11th Asian conference on Computer Vision - Volume Part II
Sorting JPEG images at a glance

Proceedings of the 4th ACM Multimedia Systems Conference
Improving tag-based image search by using linked open data

Proceedings of the 10th Conference on Open Research Areas in Information Retrieval
Exploring mapping-based visualisations of large remote image databases

Proceedings of the 6th International Symposium on Visual Information Communication and Interaction
Robust image annotation via simultaneous feature and sample outlier pursuit

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)
GLocal structural feature selection with sparsity for multimedia data understanding

Proceedings of the 21st ACM international conference on Multimedia
Fast image/video collection summarization with local clustering

Proceedings of the 21st ACM international conference on Multimedia
LIRE: open source image retrieval in Java

Proceedings of the 21st ACM international conference on Multimedia
Latent feature learning in social media network

Proceedings of the 21st ACM international conference on Multimedia
Learning user's intent using user tags: intelligent interactive image search system

Proceedings of the sixth international workshop on Exploiting semantic annotations in information retrieval
Learning semantic concepts from image database with hybrid generative/discriminative approach

Engineering Applications of Artificial Intelligence
Multimedia search reranking: A literature survey

ACM Computing Surveys (CSUR)
Co-regularized ensemble for feature selection

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Active learning with multi-label SVM classification

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Learning descriptive visual representation by semantic regularized matrix factorization

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Personalized tag recommendation based on generalized rules

ACM Transactions on Intelligent Systems and Technology (TIST) - Special Section on Intelligent Mobile Knowledge Discovery and Management Systems and Special Issue on Social Web Mining
World-wide scale geotagged image dataset for automatic image annotation and reverse geotagging

Proceedings of the 5th ACM Multimedia Systems Conference
Fashion 10000: an enriched social image dataset for fashion and clothing

Proceedings of the 5th ACM Multimedia Systems Conference

Quantified Score

Hi-index	0.00

Visualization

Abstract

In most well known image retrieval test sets, the imagery typically cannot be freely distributed or is not representative of a large community of users. In this paper we present a collection for the MIR community comprising 25000 images from the Flickr website which are redistributable for research purposes and represent a real community of users both in the image content and image tags. We have extracted the tags and EXIF image metadata, and also make all of these publicly available. In addition we discuss several challenges for benchmarking retrieval and classification methods.