Measures for benchmarking semantic web service matchmaking correctness

Authors:
Ulrich Küster;Birgitta König-Ries
Affiliations:
Institute of Computer Science, Friedrich-Schiller-University Jena, Jena, Germany;Institute of Computer Science, Friedrich-Schiller-University Jena, Jena, Germany
Venue:
ESWC'10 Proceedings of the 7th international conference on The Semantic Web: research and Applications - Volume Part II
Year:
2010

Citing 10
Cited 2

Evaluation by highly relevant documents

Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Modern Information Retrieval

Modern Information Retrieval
Cumulated gain-based evaluation of IR techniques

ACM Transactions on Information Systems (TOIS)
Learning to rank using gradient descent

ICML '05 Proceedings of the 22nd international conference on Machine learning
On the Evaluation of Semantic Web Service Matchmaking Systems

ECOWS '06 Proceedings of the European Conference on Web Services
On the reliability of information retrieval metrics based on graded relevance

Information Processing and Management: an International Journal - Special issue: AIRS2005: Information retrieval research in Asia
On rank correlation in information retrieval evaluation

ACM SIGIR Forum
Relevance Judgments for Web Services Retrieval - A Methodology and Test Collection for SWS Discovery Evaluation

ECOWS '09 Proceedings of the 2009 Seventh IEEE European Conference on Web Services
Ranking the NTCIR systems based on multigrade relevance

AIRS'04 Proceedings of the 2004 international conference on Asian Information Retrieval Technology
A classification of IR effectiveness metrics

ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval

A framework for evaluating semantic annotations of Web services: A network theory based approach for measuring annotation quality

Web Intelligence and Agent Systems
Fully automated resource retrieval in telecommunications and internet converged environments

Information Systems Frontiers

Quantified Score

Hi-index	0.00

Visualization

Abstract

Semantic Web Services (SWS) promise to take service oriented computing to a new level by allowing to semi-automate time-consuming programming tasks. At the core of SWS are solutions to the problem of SWS matchmaking, i.e., the problem of filtering and ranking a set of services with respect to a service query. Comparative evaluations of different approaches to this problem form the base for future progress in this area. Reliable evaluations require informed choices of evaluation measures and parameters. This paper establishes a solid foundation for such choices by providing a systematic discussion of the characteristics and behavior of various retrieval correctness measures in theory and through experimentation.