Efficient construction of large test collections
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Workshop on patent retrieval SIGIR 2000 workshop report
ACM SIGIR Forum
Ranking retrieval systems without relevance judgments
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Methods for ranking information retrieval systems without relevance judgments
Proceedings of the 2003 ACM symposium on Applied computing
Report on the patent retrieval task at NTCIR workshop 3
ACM SIGIR Forum
Automatic ranking of information retrieval systems using data fusion
Information Processing and Management: an International Journal
A simple and efficient sampling method for estimating AP and NDCG
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Crowdsourcing for relevance evaluation
ACM SIGIR Forum
CLEF-IP 2009: retrieval experiments in the intellectual property domain
CLEF'09 Proceedings of the 10th cross-language evaluation forum conference on Multilingual information access evaluation: text retrieval experiments
Creating a test collection: relevance judgements of cited & non-cited papers
Large Scale Semantic Access to Content (Text, Image, Video, and Sound)
A case for automatic system evaluation
ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
Effects of language and topic size in patent IR: an empirical study
CLEF'12 Proceedings of the Third international conference on Information Access Evaluation: multilinguality, multimodality, and visual analytics
| Hi-index | 0.00 |
This paper studies the use of patent citations provided by national or international patent offices as relevance judgment sources in patent test collections for IR methods. We explore a variety of ways to generate relevance judgments from patent citations and search reports. We compare them to existing automatic evaluation methods, as well as with manual evaluation results for two test collections: CLEF-IP 2009 and TREC-CHEM 2009. We observe that evaluations using patent citation based judgments, despite inherent problems (a particular kind of documents, incompleteness of judgments), incur similar manual effort as evaluations using generic automatic judgments, but provide results that are more correlated with evaluations based on manual assessments.