SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Introduction to Information Retrieval
Introduction to Information Retrieval
Focused Access to XML Documents
Overview of the CLEF-2007 Cross-Language Speech Retrieval Track
Advances in Multilingual and Multimodal Information Retrieval
Information retrieval test collection for searching spontaneous Czech speech
TSD'07 Proceedings of the 10th international conference on Text, speech and dialogue
New metrics for meaningful evaluation of informally structured speech retrieval
ECIR'12 Proceedings of the 34th European conference on Advances in Information Retrieval
Overview of the CLEF-2006 cross-language speech retrieval track
CLEF'06 Proceedings of the 7th international conference on Cross-Language Evaluation Forum: evaluation of multilingual and multi-modal information retrieval
| Hi-index | 0.00 |
This paper deals with evaluation of information retrieval from unsegmented speech. We focus on Mean Generalized Average Precision, the evaluation measure widely used for unsegmented speech retrieval. This measure is designed to allow certain tolerance in matching retrieval results (starting points of relevant segments) against a gold standard relevance assessment. It employs a Penalty Function which evaluates non-exact matches in the retrieval results based on their distance from the beginnings of their nearest true relevant segments. However, the choice of the Penalty Function is usually ad-hoc and does not necessary reflect users' perception of the speech retrieval quality. We perform a lab test to study satisfaction of users of a speech retrieval system to empirically estimate the optimal shape of the Penalty Function.