Finite Automata for Compact Representation of Language Models in NLP

  • Authors:
  • Jan Daciuk;Gertjan van Noord

  • Affiliations:
  • -;-

  • Venue:
  • CIAA '01 Revised Papers from the 6th International Conference on Implementation and Application of Automata
  • Year:
  • 2001

Quantified Score

Hi-index 0.00

Visualization

Abstract

A technique for compact representation of language models in Natural Language Processing is presented. After a brief review of the motivations for a more compact representation of such language models, it is shown how finite-state automata can be used to compactly represent such language models. The technique can be seen as an application and extension of perfect hashing by means of finite-state automata. Preliminary practical experiments indicate that the technique yields considerable and important space savings of up to 90% in practice.