A New Measure of the Cluster Hypothesis

  • Authors:
  • Mark D. Smucker;James Allan

  • Affiliations:
  • Department of Management Sciences, University of Waterloo,;Center for Intelligent Information Retrieval Department of Computer Science, University of Massachusetts, Amherst

  • Venue:
  • ICTIR '09 Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

We have found that the nearest neighbor (NN) test is an insufficient measure of the cluster hypothesis. The NN test is a local measure of the cluster hypothesis. Designers of new document-to-document similarity measures may incorrectly report effective clustering of relevant documents if they use the NN test alone. Utilizing a measure from network analysis, we present a new, global measure of the cluster hypothesis: normalized mean reciprocal distance. When used together with a local measure, such as the NN test, this new global measure allows researchers to better measure the cluster hypothesis.