Parsing with Context-Free Grammars and Word Statistics

  • Authors:
  • Eugene Charniak

  • Affiliations:
  • -

  • Venue:
  • Parsing with Context-Free Grammars and Word Statistics
  • Year:
  • 1995

Quantified Score

Hi-index 0.00

Visualization

Abstract

We present a language model in which the probability of a sentence is the sum of the individual parse probabilities, and these are calculated using a probabilistic context-free grammar (PCFG) plus statistics on individual words and how they fit into parses. We have used the model to improve syntactic disambiguation. After training on Wall Street Journal (WSJ) text we tested on about 200 WSJ sentence restricted to the 5400 most common words from our training. We observed a 41\ performance of our PCFG without the use of the word statistics.