A formal basis for performance evaluation of natural language understanding systems
Computational Linguistics
Operating statistics for the transformational question answering system
Computational Linguistics
Natural language for expert systems: comparisons with database systems
ACL '84 Proceedings of the 10th International Conference on Computational Linguistics and 22nd annual meeting on Association for Computational Linguistics
The structure of user-adviser dialogues: is there method in their madness?
ACL '86 Proceedings of the 24th annual meeting on Association for Computational Linguistics
Linguistic analysis of natural language communication with computers
COLING '80 Proceedings of the 8th conference on Computational linguistics
Evaluating natural language systems: a sourcebook approach
COLING '88 Proceedings of the 12th conference on Computational linguistics - Volume 2
What can I say?: evaluating a spoken language interface to Email
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Empirical studies in discourse
Computational Linguistics
Computational Linguistics
Evaluating discourse processing algorithms
ACL '89 Proceedings of the 27th annual meeting on Association for Computational Linguistics
A case study of natural language customisation: the practical effects of world knowledge
COLING '92 Proceedings of the 14th conference on Computational linguistics - Volume 2
Performance measures for the next generation of spoken natural language dialog systems
ISDS '97 Interactive Spoken Dialog Systems on Bringing Speech and NLP Together in Real Applications
| Hi-index | 0.00 |
This paper presents a critical discussion of the various approaches that have been used in the evaluation of Natural Language systems. We conclude that previous approaches have neglected to evaluate systems in the context of their use, e.g. solving a task requiring data retrieval. This raises questions about the validity of such approaches. In the second half of the paper, we report a laboratory study using the Wizard of Oz technique to identify NL requirements for carrying out this task. We evaluate the demands that task dialogues collected using this technique, place upon a prototype Natural Language system. We identify three important requirements which arose from the task that we gave our subjects: operators specific to the task of database access, complex contextual reference and reference to the structure of the information source. We discuss how these might be satisfied by future Natural Language systems.