Exploring semi-automatic nugget extraction for Japanese one click access evaluation

Authors:
Matthew Ekstrand-Abueg;Virgil Pavlu;Makoto Kato;Tetsuya Sakai;Takehiro Yamamoto;Mayu Iwata
Affiliations:
Northeastern University, Boston, MA, USA;Northeastern University, Boston, MA, USA;Kyoto University, Kyoto, Japan;Microsoft Research Asia, Beijing, China;Kyoto University, Kyoto, Japan;Osaka University, Osaka, Japan
Venue:
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Year:
2013

Citing 7
Cited 1

A decision-theoretic generalization of on-line learning and an application to boosting

EuroCOLT '95 Proceedings of the Second European Conference on Computational Learning Theory
Methods for automatically evaluating answers to complex questions

Information Retrieval
Novelty and diversity in information retrieval evaluation

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Good abandonment in mobile and PC internet search

Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Click the search button and be happy: evaluating direct and immediate information access

Proceedings of the 20th ACM international conference on Information and knowledge management
Frontiers, challenges, and opportunities for information retrieval: Report from SWIRL 2012 the second strategic workshop on information retrieval in Lorne

ACM SIGIR Forum
Constructing test collections by inferring document relevance via extracted relevant information

Proceedings of the 21st ACM international conference on Information and knowledge management

Report from the NTCIR-10 1CLICK-2 Japanese subtask: baselines, upperbounds and evaluation robustness

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Quantified Score

Hi-index	0.00

Visualization

Abstract

Building test collections based on nuggets is useful evaluating systems that return documents, answers, or summaries. However, nugget construction requires a lot of manual work and is not feasible for large query sets. Towards an efficient and scalable nugget-based evaluation, we study the applicability of semi-automatic nugget extraction in the context of the ongoing NTCIR One Click Access (1CLICK) task. We compare manually-extracted and semi-automatically-extracted Japanese nuggets to demonstrate the coverage and efficiency of the semi-automatic nugget extraction. Our findings suggest that the manual nugget extraction can be replaced with a direct adaptation of the English semi-automatic nugget extraction system, especially for queries for which the user desires broad answers from free-form text.