Content-based query of image databases: inspirations from text retrieval
Pattern Recognition Letters - Selected papers from the 11th scandinavian conference on image analysis
A web-based evaluation system for CBIR
MULTIMEDIA '01 Proceedings of the 2001 ACM workshops on Multimedia: multimedia information retrieval
Scientific workflow management and the Kepler system: Research Articles
Concurrency and Computation: Practice & Experience - Workflow in Grid Systems
Computer
The challenge problem for automated detection of 101 semantic concepts in multimedia
MULTIMEDIA '06 Proceedings of the 14th annual ACM international conference on Multimedia
On the history of evaluation in IR
Journal of Information Science
Crowdsourcing for relevance evaluation
ACM SIGIR Forum
Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science
Improvements that don't add up: ad-hoc retrieval results since 1998
Proceedings of the 18th ACM conference on Information and knowledge management
The visual concept detection task in ImageCLEF 2008
CLEF'08 Proceedings of the 9th Cross-language evaluation forum conference on Evaluating systems for multilingual and multimodal information access
CLEF 2009: Grid@CLEF pilot track overview
CLEF'09 Proceedings of the 10th cross-language evaluation forum conference on Multilingual information access evaluation: text retrieval experiments
A large-scale system evaluation on component-level
ECIR'11 Proceedings of the 33rd European conference on Advances in information retrieval
A tool for comparative IR evaluation on component level
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Comparing IR system components using beanplots
CLEF'12 Proceedings of the Third international conference on Information Access Evaluation: multilinguality, multimodality, and visual analytics
| Hi-index | 0.00 |
Automated component-level evaluation of information retrieval (IR) is the main focus of this paper. We present a review of the current state of web-based and component-level evaluation. Based on these systems, propositions are made for a comprehensive framework for web service-based component-level IR system evaluation. The advantages of such an approach are considered, as well as the requirements for implementing it. Acceptance of such systems by researchers who develop components and systems is crucial for having an impact and requires that a clear benefit is demonstrated.