Information theoretic learning with adaptive kernels

Authors:
Abhishek Singh;José C. Príncipe
Affiliations:
Computational NeuroEngineering Laboratory, NEB 486, Bldg #33, P.O. Box 116130, University of Florida, Gainesville, FL 32611, USA;Computational NeuroEngineering Laboratory, NEB 486, Bldg #33, P.O. Box 116130, University of Florida, Gainesville, FL 32611, USA
Venue:
Signal Processing
Year:
2011

Citing 8
Cited 4

An information-maximization approach to blind separation and blind deconvolution

Neural Computation
Topographic Independent Component Analysis

Neural Computation
A Method for Selecting the Bin Size of a Time Histogram

Neural Computation
On the Choice of Smoothing Parameters for Parzen Estimators of Probability Density Functions

IEEE Transactions on Computers
An error-entropy minimization algorithm for supervised training ofnonlinear adaptive systems

IEEE Transactions on Signal Processing
The least mean fourth (LMF) adaptive algorithm and its family

IEEE Transactions on Information Theory
Generalized information potential criterion for adaptive system training

IEEE Transactions on Neural Networks
Advanced search algorithms for information-theoretic learning with kernel-based estimators

IEEE Transactions on Neural Networks

The automatic model selection and variable kernel width for RBF neural networks

Neurocomputing
Fixed budget quantized kernel least-mean-square algorithm

Signal Processing
The C-loss function for pattern classification

Pattern Recognition
Regularized discriminant entropy analysis

Pattern Recognition

Quantified Score

Hi-index	0.08

Visualization

Abstract

This paper presents an online algorithm for adapting the kernel width that is a free parameter in information theoretic cost functions using Renyi's entropy. This kernel computes the interactions between the error samples and essentially controls the nature of the performance surface over which the parameters of the system adapt. Since the error in an adaptive system is non-stationary during training, a fixed value of the kernel width may affect the adaptation dynamics and even compromise the location of the global optimum in parameter space. The proposed online algorithm for adapting the kernel width is derived from first principles and minimizes the Kullback-Leibler divergence between the estimated error density and the true density. We characterize the performance of this novel approach with simulations of linear and nonlinear systems training, using the minimum error entropy criterion with the proposed adaptive kernel algorithm. We conclude that adapting the kernel width improves the rate of convergence of the parameters, and decouples the convergence rate and misadjustment of the filter weights.