Towards a theory of natural language interfaces to databases
Proceedings of the 8th international conference on Intelligent user interfaces
Using Multiple Clause Constructors in Inductive Logic Programming for Semantic Parsing
EMCL '01 Proceedings of the 12th European Conference on Machine Learning
Swoogle: a search and metadata engine for the semantic web
Proceedings of the thirteenth ACM international conference on Information and knowledge management
Mining Software Repositories with iSPAROL and a Software Evolution Ontology
MSR '07 Proceedings of the Fourth International Workshop on Mining Software Repositories
Quality of bug reports in Eclipse
Proceedings of the 2007 OOPSLA workshop on eclipse technology eXchange
How useful are natural language interfaces to the semantic web for casual end-users?
ISWC'07/ASWC'07 Proceedings of the 6th international The semantic web and 2nd Asian conference on Asian semantic web conference
Hybrid search: effectively combining keywords and semantic searches
ESWC'08 Proceedings of the 5th European semantic web conference on The semantic web: research and applications
Querying ontologies: a controlled english interface for end-users
ISWC'05 Proceedings of the 4th international conference on The Semantic Web
GINO – a guided input natural language ontology editor
ISWC'06 Proceedings of the 5th international conference on The Semantic Web
SemSearch: a search engine for the semantic web
EKAW'06 Proceedings of the 15th international conference on Managing Knowledge in a World of Networks
Semantic need: guiding metadata annotations by questions people #ask
ISWC'10 Proceedings of the 9th international semantic web conference on The semantic web - Volume Part I
Effective and efficient entity search in RDF data
ISWC'11 Proceedings of the 10th international conference on The semantic web - Volume Part I
Infrastructure and workflow for the formal evaluation of semantic search technologies
Proceedings of the 2011 workshop on Data infrastructurEs for supporting information retrieval evaluation
Semantic evaluation at large scale (SEALS)
Proceedings of the 21st international conference companion on World Wide Web
Semantic query answering in digital repositories: Semantic Search v2 for DSpace
International Journal of Metadata, Semantics and Ontologies
Hi-index | 0.02 |
The main problem with the state of the art in the semantic search domain is the lack of comprehensive evaluations. There exist only a few efforts to evaluate semantic search tools and to compare the results with other evaluations of their kind. In this paper, we present a systematic approach for testing and benchmarking semantic search tools that was developed within the SEALS project. Unlike other semantic web evaluations our methodology tests search tools both automatically and interactively with a human user in the loop. This allows us to test not only functional performance measures, such as precision and recall, but also usability issues, such as ease of use and comprehensibility of the query language. The paper describes the evaluation goals and assumptions; the criteria and metrics; the type of experiments we will conduct as well as the datasets required to conduct the evaluation in the context of the SEALS initiative. To our knowledge it is the first effort to present a comprehensive evaluation methodology for Semantic Web search tools.