Learning from data with localized regression and differential evolution

• Authors: Mark A. Buckner;Robert E. Uhrig
• Venue: Learning from data with localized regression and differential evolution
• Year: 2003

Learning from data is fast becoming the rule rather than the exception for many science and engineering research problems, particularly those encountered in nuclear engineering. Problems associated with learning from data fall under the more general category of inverse problems . A data-drive inverse problem involves constructing a predictive model of a target system from a collection of input/output observations. One of the difficulties associated with constructing a model that approximates such unknown causes based solely on observations of their effects is that collinearities in the input data result in the problem being ill-posed. Ill-posed problems cause models obtained by conventional techniques, such as linear regression, neural networks and kernel techniques, to become unstable, producing unreliable results. Methods of regularization using ordinary ridge regression (ORR) and kernel regression (KR) have been proposed as viable solutions to ill-posed problems. Successful application of ORR and KR require the selection of optimal parameter values—ridge parameters for ORR and bandwidth parameters for KR. The common practice for both methods is to select a single parameter based on minimizing an objective function which is an estimate of empirical risk. The single parameter value is then applied to all predictor variables indiscriminately, in a sort of one-size-fits-all fashion. Versions of ORR and KR have been proposed that make use of individual localized ridge and a matrix of localized bandwidth parameters that are optimally selected based on the relevance of their associated predictor variables to reducing empirical risk. While the practical and theoretical value of both localized regression techniques is recognized they have obtained limited use because of the difficulties associated with selecting multiple optimal ridge parameters for localized ridge regression (LRR)—defined as the localized ridge regression problem—and multiple optimal bandwidth parameters for localized kernel regression (LKR)—defined as the localized kernel regression problem—particularly for multivariate predictor data with more than four variables. This dissertation introduces a method of selecting optimal ridge parameters for LRR and a method of selecting a matrix of optimal bandwidth parameters for LKR based on the use of Differential Evolution (DE), a population based direct search global optimization technique. Three different objective functions, selected as prediction risk estimators, were developed and evaluated for LRR: Mallows' CL, an Information Complexity (ICOMP) based method of regression parameter selection (ICOMPRPS), and Generalized Cross-Validation (GCV). Leave-one-out cross-validation (LOO-CV) was used as the objective function for LKR. (Abstract shortened by UMI.)