Evaluating message understanding systems: an analysis of the third message understanding conference (MUC-3)

Authors:
Nancy Chinchor;David D. Lewis;Lynette Hirschman
Affiliations:
Science Applications International Corp.;University of Chicago;Massachusetts Institute of Technology
Venue:
Computational Linguistics
Year:
1993

Citing 12
Cited 38

Workshop on the evaluation of natural language processing systems

Computational Linguistics
Procedure for quantitatively comparing the syntactic coverage of English grammars

HLT '91 Proceedings of the workshop on Speech and Natural Language
An evaluation of text analysis technologies

AI Magazine
Information Retrieval

Information Retrieval
Comparing MUCK-II and MUC-3: assessing the difficulty of different tasks

MUC3 '91 Proceedings of the 3rd conference on Message understanding
MUC-3 linguistic phenomena test experiment

MUC3 '91 Proceedings of the 3rd conference on Message understanding
Data extraction as text categorization: an experiment with the MUC-3 corpus

MUC3 '91 Proceedings of the 3rd conference on Message understanding
The statistical significance of the MUC-4 results

MUC4 '92 Proceedings of the 4th conference on Message understanding
Text filtering in MUC-3 and MUC-4

MUC4 '92 Proceedings of the 4th conference on Message understanding
An adjunct test for discourse processing in MUC-4

MUC4 '92 Proceedings of the 4th conference on Message understanding
GE adjunct test report: object-oriented design and scoring for MUC-4

MUC4 '92 Proceedings of the 4th conference on Message understanding
Subject-based evaluation measures for interactive spoken language systems

HLT '91 Proceedings of the workshop on Speech and Natural Language

Abstracting of legal cases: the SALOMON experience

Proceedings of the 6th international conference on Artificial intelligence and law
Concept-based knowledge discovery in texts extracted from the Web

ACM SIGKDD Explorations Newsletter
Customizing information: Part 1, Getting what we need, when we need it

Computer
Can We Make Information Extraction More Adaptive?

Information Extraction: Towards Scalable, Adaptable Systems
Empirical studies in discourse

Computational Linguistics
An empirical assessment of semantic interpretation

NAACL 2000 Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference
An information extraction core system for real world German text processing

ANLC '97 Proceedings of the fifth conference on Applied natural language processing
Surprise! What's in a Cebuano or Hindi Name?

ACM Transactions on Asian Language Information Processing (TALIP)
Reference resolution using semantic patterns in Japanese newspaper articles

COLING '94 Proceedings of the 15th conference on Computational linguistics - Volume 2
More accurate tests for the statistical significance of result differences

COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 2
TEG: a hybrid approach to information extraction

Proceedings of the thirteenth ACM international conference on Information and knowledge management
Hybrid semantic tagging for information extraction

WWW '05 Special interest tracks and posters of the 14th international conference on World Wide Web
The statistical significance of the MUC-5 results

MUC5 '93 Proceedings of the 5th conference on Message understanding
The statistical significance of the MUC-4 results

MUC4 '92 Proceedings of the 4th conference on Message understanding
Statistical significance of MUC-6 results

MUC6 '95 Proceedings of the 6th conference on Message understanding
Survey of the Message Understanding Conferences

HLT '93 Proceedings of the workshop on Human Language Technology
MUC/MET evaluation trends

TIPSTER '98 Proceedings of a workshop on held at Baltimore, Maryland: October 13-15, 1998
Playing the telephone game: determining the hierarchical structure of perspective and speech expressions

COLING '04 Proceedings of the 20th international conference on Computational Linguistics
URES: an unsupervised web relation extraction system

COLING-ACL '06 Proceedings of the COLING/ACL on Main conference poster sessions
Promoting Insight-Based Evaluation of Visualizations: From Contest to Benchmark Repository

IEEE Transactions on Visualization and Computer Graphics
Wide-coverage deep statistical parsing using automatic dependency structure annotation

Computational Linguistics
Creating realistic, scenario-based synthetic data for test and evaluation of information analytics software

Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization
Determining termhood for learning domain ontologies in a probabilistic framework

AusDM '07 Proceedings of the sixth Australasian conference on Data mining and analytics - Volume 70
The Value of Information Visualization

Information Visualization
Web-scale named entity recognition

Proceedings of the 17th ACM conference on Information and knowledge management
Orthographic co-reference resolution between proper nouns through the calculation of the relation of "replicancia"

CorefApp '99 Proceedings of the Workshop on Coreference and its Applications
A probabilistic framework for automatic term recognition

Intelligent Data Analysis
Learning document-level semantic properties from free-text annotations

Journal of Artificial Intelligence Research
The NIST 2008 Metrics for machine translation challenge--overview, methodology, metrics, and results

Machine Translation
Formal and functional assessment of the pyramid method for summary content evaluation*

Natural Language Engineering
Cause identification from aviation safety incident reports via weakly supervised semantic lexicon construction

Journal of Artificial Intelligence Research
Template-based information extraction without the templates

HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Active learning with Amazon Mechanical Turk

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Evaluating web search result summaries

ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval
Multi event extraction guided by global constraints

NAACL HLT '12 Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Measuring the use of factual information in test-taker essays

Proceedings of the Seventh Workshop on Building Educational Applications Using NLP
Semantic role labeling of implicit arguments for nominal predicates

Computational Linguistics
Information extraction as a filtering task

Proceedings of the 22nd ACM international conference on Conference on information & knowledge management

Quantified Score

Hi-index	0.01

Visualization

Abstract

This paper describes and analyzes the results of the Third Message Understanding Conference (MUC-3). It reviews the purpose, history, and methodology of the conference, summarizes the participating systems, discusses issues of measuring system effectiveness, describes the linguistic phenomena tests, and provides a critical look at the evaluation in terms of the lessons learned. One of the common problems with evaluations is that the statistical significance of the results is unknown. In the discussion of system performance, the statistical significance of the evaluation results is reported and the use of approximate randomization to calculate the statistical significance of the results of MUC-3 is described.