Designing testsuites for grammar-based systems in applications

Authors:
Valeria de Paiva;Tracy Holloway King
Affiliations:
Palo Alto Research Center, Palo Alto, CA;Palo Alto Research Center, Palo Alto, CA
Venue:
GEAF '08 Proceedings of the Workshop on Grammar Engineering Across Frameworks
Year:
2008

Citing 4
Cited 3

Building a question answering test collection

SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
TSNLP: Test Suites for Natural Language Processing

COLING '96 Proceedings of the 16th conference on Computational linguistics - Volume 2
Type-checking in formally non-typed systems

SETQA-NLP '08 Software Engineering, Testing, and Quality Assurance for Natural Language Processing
Software testing and the naturally occurring data assumption in natural language processing

SETQA-NLP '08 Software Engineering, Testing, and Quality Assurance for Natural Language Processing

Context-dependent regression testing for natural language processing

SETQA-NLP '09 Proceedings of the Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing
Modular resource development and diagnostic evaluation framework for fast NLP system improvement

SETQA-NLP '09 Proceedings of the Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing
Developing German semantics on the basis of parallel LFG grammars

GEAF '09 Proceedings of the 2009 Workshop on Grammar Engineering Across Frameworks

Quantified Score

Hi-index	0.00

Visualization

Abstract

In complex grammar-based systems, even small changes may have an unforeseeable impact on overall system performance. Regression testing of the system and its components becomes crucial for the grammar engineers developing the system. As part of this regression testing, the testsuites themselves must be designed to accurately assess coverage and progress and to help rapidly identify problems. We describe a system of passage-query pairs divided into three types of phenomenon-based test-suites (sanity, query, basic correct). These allow for rapid development and for specific coverage assessment. In addition, real-world testsuites allow for overall performance and coverage assessment. These testsuites are used in conjunction with the more traditional representation-based regression testsuites used by grammar engineers.