Learning with sample dependent hypothesis spaces

  • Authors:
  • Qiang Wu;Ding-Xuan Zhou

  • Affiliations:
  • Department of Statistical Science, Institute of Genome Sciences and Policy, Duke University, Durham, NC 27708, USA;Department of Mathematics, City University of Hong Kong, Tat Chee Avenue, Kowloon, Hong Kong

  • Venue:
  • Computers & Mathematics with Applications
  • Year:
  • 2008

Quantified Score

Hi-index 0.09

Visualization

Abstract

Many learning algorithms use hypothesis spaces which are trained from samples, but little theoretical work has been devoted to the study of these algorithms. In this paper we show that mathematical analysis for these algorithms is essentially different from that for algorithms with hypothesis spaces independent of the sample or depending only on the sample size. The difficulty lies in the lack of a proper characterization of approximation error. To overcome this difficulty, we propose an idea of using a larger function class (not necessarily linear space) containing the union of all possible hypothesis spaces (varying with the sample) to measure the approximation ability of the algorithm. We show how this idea provides error analysis for two particular classes of learning algorithms in kernel methods: learning the kernel via regularization and coefficient based regularization. We demonstrate the power of this approach by its wide applicability.