New metrics for meaningful evaluation of informally structured speech retrieval

Authors:
Maria Eskevich;Walid Magdy;Gareth J. F. Jones
Affiliations:
Centre for Digital Video Processing, School of Computing, Dublin City University, Dublin, Ireland;Centre for Next Generation Localisation, School of Computing, Dublin City University, Dublin, Ireland;Centre for Digital Video Processing, School of Computing, Dublin City University, Dublin, Ireland and Centre for Next Generation Localisation, School of Computing, Dublin City University, Dublin, ...
Venue:
ECIR'12 Proceedings of the 34th European conference on Advances in Information Retrieval
Year:
2012

Citing 7
Cited 4

Using graded relevance assessments in IR evaluation

Journal of the American Society for Information Science and Technology
TextTiling: segmenting text into multi-paragraph subtopic passages

Computational Linguistics
Advances in domain independent linear text segmentation

NAACL 2000 Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference
One-sided measures for evaluating ranked retrieval effectiveness with spontaneous conversational speech

SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
INEX 2007 Evaluation Measures

Focused Access to XML Documents
Overview of the CLEF-2007 Cross-Language Speech Retrieval Track

Advances in Multilingual and Multimodal Information Retrieval
Information Retrieval: Implementing and Evaluating Search Engines

Information Retrieval: Implementing and Evaluating Search Engines

Penalty functions for evaluation measures of unsegmented speech retrieval

CLEF'12 Proceedings of the Third international conference on Information Access Evaluation: multilinguality, multimodality, and visual analytics
Multimedia information seeking through search and hyperlinking

Proceedings of the 3rd ACM conference on International conference on multimedia retrieval
An LDA-smoothed relevance model for document expansion: a case study for spoken document retrieval

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Segmentation strategies for passage retrieval in audio-visual documents

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Quantified Score

Hi-index	0.00

Visualization

Abstract

Search effectiveness for tasks where the retrieval units are clearly defined documents is generally evaluated using standard measures such as mean average precision (MAP). However, many practical speech search tasks focus on content within large spoken files lacking defined structure. These data must be segmented into smaller units for search which may only partially overlap with relevant material. We introduce two new metrics for the evaluation of search effectiveness for informally structured speech data: mean average segment precision (MASP) which measures retrieval performance in terms of both content segmentation and ranking with respect to relevance; and mean average segment distance-weighted precision (MASDWP) which takes into account the distance between the start of the relevant segment and the retrieved segment. We demonstrate the effectiveness of these new metrics on a retrieval test collection based on the AMI meeting corpus.