The differogram: Non-parametric noise variance estimation and its use for model selection

Authors:
Kristiaan Pelckmans;Jos De Brabanter;Johan A. K. Suykens;Bart De Moor
Affiliations:
K.U. Leuven, ESAT - SCD/SISTA, Kasteelpark Arenberg 10, B-3001 Leuven (Heverlee), Belgium;K.U. Leuven, ESAT - SCD/SISTA, Kasteelpark Arenberg 10, B-3001 Leuven (Heverlee), Belgium and Hogeschool KaHo Sint-Lieven (Associatie KULeuven), Departement Industrieel Ingenieur, Belgium;K.U. Leuven, ESAT - SCD/SISTA, Kasteelpark Arenberg 10, B-3001 Leuven (Heverlee), Belgium;K.U. Leuven, ESAT - SCD/SISTA, Kasteelpark Arenberg 10, B-3001 Leuven (Heverlee), Belgium
Venue:
Neurocomputing
Year:
2005

Citing 7
Cited 5

Solving Ill-Conditioned and Singular Linear Systems: A Tutorial on Regularization

SIAM Review
Neural Networks for Pattern Recognition

Neural Networks for Pattern Recognition
Choosing Multiple Parameters for Support Vector Machines

Machine Learning
Robust Cross-Validation Score Function for Non-linear Function Estimation

ICANN '02 Proceedings of the International Conference on Artificial Neural Networks
Practical selection of SVM parameters and noise estimation for SVM regression

Neural Networks
The evidence framework applied to classification networks

Neural Computation
Financial time series prediction using least squares support vector machines within the evidence framework

IEEE Transactions on Neural Networks

The Concentration of Fractional Distances

IEEE Transactions on Knowledge and Data Engineering
On Nonparametric Residual Variance Estimation

Neural Processing Letters
Residual variance estimation in machine learning

Neurocomputing
Regularized Discriminant Analysis, Ridge Regression and Beyond

The Journal of Machine Learning Research
Optimized Parameter Search for Large Datasets of the Regularization Parameter and Feature Selection for Ridge Regression

Neural Processing Letters

Quantified Score

Hi-index	0.01

Visualization

Abstract

Model-free estimates of the noise variance are important in model selection and setting tuning parameters. In this paper a data representation is discussed which leads to such an estimator suitable for multivariate data. Its visual representation-called the differogram cloud here-is based on the 2-norm of the differences of input and output data. The crucial concept of locality in this representation is translated as the increasing variance of the difference, which does not rely explicitly on an extra hyper-parameter. Connections with U-statistics, Taylor series expansions and other related methods are given. Numerical simulations indicate a convergence of the estimator. This paper extends results towards a time-dependent setting and to the case of non-Gaussian noise models or outliers. As an application, this paper focuses on model selection for Least Squares Support Vector Machines. For this purpose, a variant of the LS-SVM regressor is derived based on Morozov's discrepancy principle relating the regularization constant directly with the (observed) noise level.