Matching utterances to rich knowledge structures to acquire a model of the speaker's goal
Proceedings of the 3rd international conference on Knowledge capture
Using decision trees for conference resolution
IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2
Evaluating web search result summaries
ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval
| Hi-index | 0.00 |
The MUC-3 evaluation metrics are measures of performance for the MUC-3 template fill task. Obtaining summary measures of performance necessitates the loss of information about many details of performance. The utility of summary measures for comparison of performance over time and across systems should outweigh this loss of detail. The template fill task is complex because of the varying nature of the fills for each slot and the interdependencies of the slots. The evaluation metrics used in MUC-3 were adapted from traditional measures in information retrieval and signal procesing and were still evolving to fit the more complex data extraction task of MUC-3 when the evaluation was performed. The scoring of the template fill task and the calculation of the metrics used in MUC-3 will be described here. This description is meant to assist in the analysis of the MUC-3 results and in the further evolution of the evaluation metrics.