Estimation by the nearest neighbor rule

• Authors: T. Cover
• Venue: IEEE Transactions on Information Theory
• Year: 2006

LetR^{ast}denote the Bayes risk (minimum expected loss) for the problem of estimatingtheta varepsilon Theta, given an observed random variablex, joint probability distributionF(x,theta), and loss functionL. Consider the problem in which the only knowledge ofFis that which can be inferred from samples(x_{1},theta_{1}),(x_{2},theta_{2}), cdots ,(x_{n}, theta_{n}), where the(x_{i}, theta_{i})'sare independently identically distributed according toF. Let the nearest neighbor estimate of the parameterthetaassociated with an observationxbe defined to be the parametertheta_{n}^{'}associated with the nearest neighborx_{n}^{'}tox. Let R be the large sample risk of the nearest neighbor rule. It will be shown, for a wide range of probability distributions, thatR leq 2R^{ast}for metric loss functions andR = 2R^{ast}for squared-error loss functions. A simple estimator using the nearestkneighbors yieldsR = R^{ast} (1 + 1/k)in the squared-error loss case. In this sense, it can be said that at least haft the information in the infinite training set is contained in the nearest neighbor. This paper is an extension of earlier work[q from the problem of classification by the nearest neighbor rule to that of estimation. However, the unbounded loss functions in the estimation problem introduce additional problems concerning the convergence of the unconditional risk. Thus some work is devoted to the investigation of natural conditions on the underlying distribution assuring the desired convergence.