Minimum Cross-Entropy Pattern Classification and Cluster Analysis

  • Authors:

  • John E. Shore;Robert M. Gray

  • Affiliations:

  • SENIOR MEMBER, IEEE, Information Technology Division, Naval Research Laboratory, Washington, DC 20375.;FELLOW, IEEE, Department of Electrical Engineering, Stanford University, Stanford, CA 94305.

  • Venue:
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Year:
  • 1982

Quantified Score

Hi-index 0.14

Visualization

Abstract

This paper considers the problem of classifying an input vector of measurements by a nearest neighbor rule applied to a fixed set of vectors. The fixed vectors are sometimes called characteristic feature vectors, codewords, cluster centers, models, reproductions, etc. The nearest neighbor rule considered uses a non-Euclidean information-theoretic distortion measure that is not a metric, but that nevertheless leads to a classification method that is optimal in a well-defined sense and is also computationally attractive. Furthermore, the distortion measure results in a simple method of computing cluster centroids. Our approach is based on the minimization of cross-entropy (also called discrimination information, directed divergence, K-L number), and can be viewed as a refinement of a general classification method due to Kullback. The refinement exploits special properties of cross-entropy that hold when the probability densities involved happen to be minimum cross-entropy densities. The approach is a generalization of a recently developed speech coding technique called speech coding by vector quantization.