An approach to indexing and clustering news stories using continuous language models

  • Authors:
  • Richard Bache;Fabio Crestani

  • Affiliations:
  • University of Glasgow, Glasgow, Scotland;University of Lugano, Lugano, Switzerland

  • Venue:
  • NLDB'10 Proceedings of the Natural language processing and information systems, and 15th international conference on Applications of natural language to information systems
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

Within the vocabulary used in a set of news stories a minority of terms will be topic-specific in that they occur largely or solely within those stories belonging to a common event. When applying unsupervised learning techniques such as clustering it is useful to determine which words are event-specific and which topic they relate to. Continuous language models are used to model the generation of news stories over time and from these models two measures are derived: bendiness which indicates whether a word is event specific and shape distance which indicates whether two terms are likely to relate to the same topic. These are used to construct a new clustering technique which identifies and characterises the underlying events within the news stream.