De-Biasing for Intrinsic Dimension Estimation

  • Authors:

  • Kevin M. Carter;Alfred O. Hero;Raviv Raich

  • Affiliations:

  • Department of EECS, University of Michigan, Ann Arbor, MI 48109. kmcarter@umich.edu;Department of EECS, University of Michigan, Ann Arbor, MI 48109. hero@umich.edu;Department of EECS, University of Michigan, Ann Arbor, MI 48109. ravivr@umich.edu

  • Venue:
  • SSP '07 Proceedings of the 2007 IEEE/SP 14th Workshop on Statistical Signal Processing
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

Many algorithms have been proposed for estimating the intrinsic dimension of high dimensional data. A phenomenon common to all of them is a negative bias, perceived to be the result of under-sampling. We propose improved methods for estimating intrinsic dimension, taking manifold boundaries into consideration. By estimating dimension locally, we are able to analyze and reduce the effect that sample data depth has on the negative bias. Additionally, we offer improvements to an existing algorithm for dimension estimation, based on k-nearest neighbor graphs, and offer an algorithm for adapting any dimension estimation algorithm to operate locally. Finally, we illustrate the uses of local dimension estimation with data sets consisting of multiple manifolds, including applications such as diagnosing anomalies in router networks and image segmentation.