Computer systems that learn: classification and prediction methods from statistics, neural nets, machine learning, and expert systems
Unsupervised stratification of cross-validation for accuracy estimation
Artificial Intelligence
Response Surface Methodology: Process and Product in Optimization Using Designed Experiments
Response Surface Methodology: Process and Product in Optimization Using Designed Experiments
On Comparing Classifiers: Pitfalls toAvoid and a Recommended Approach
Data Mining and Knowledge Discovery
Efficient Global Optimization of Expensive Black-Box Functions
Journal of Global Optimization
Model Selection and Error Estimation
Machine Learning
A Racing Algorithm for Configuring Metaheuristics
GECCO '02 Proceedings of the Genetic and Evolutionary Computation Conference
Inference for the Generalization Error
Machine Learning
A comprehensive survey of fitness approximation in evolutionary computation
Soft Computing - A Fusion of Foundations, Methodologies and Applications
No Unbiased Estimator of the Variance of K-Fold Cross-Validation
The Journal of Machine Learning Research
Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-Models
Journal of Global Optimization
Predictive Approaches for Choosing Hyperparameters in Gaussian Processes
Neural Computation
Experimental Research in Evolutionary Computation: The New Experimentalism (Natural Computing Series)
Resampling Methods: A Practical Guide to Data Analysis
Resampling Methods: A Practical Guide to Data Analysis
Meta-Modeling in Multiobjective Optimization
Multiobjective Optimization
Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Computational Statistics & Data Analysis
A study of cross-validation and bootstrap for accuracy estimation and model selection
IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2
Sequential Approximate Multiobjective Optimization Using Computational Intelligence
Sequential Approximate Multiobjective Optimization Using Computational Intelligence
Comparing parameter tuning methods for evolutionary algorithms
CEC'09 Proceedings of the Eleventh conference on Congress on Evolutionary Computation
Generalizing surrogate-assisted evolutionary computation
IEEE Transactions on Evolutionary Computation
Benchmarking evolutionary algorithms: towards exploratory landscape analysis
PPSN'10 Proceedings of the 11th international conference on Parallel problem solving from nature: Part I
Comparison-based optimizers need comparison-based surrogates
PPSN'10 Proceedings of the 11th international conference on Parallel problem solving from nature: Part I
On expected-improvement criteria for model-based multi-objective optimization
PPSN'10 Proceedings of the 11th international conference on Parallel problem solving from nature: Part I
Exploratory landscape analysis
Proceedings of the 13th annual conference on Genetic and evolutionary computation
Local meta-models for optimization using evolution strategies
PPSN'06 Proceedings of the 9th international conference on Parallel Problem Solving from Nature
Accelerating evolutionary algorithms with Gaussian process fitness function models
IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews
Combining Global and Local Surrogate Models to Accelerate Evolutionary Optimization
IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews
A framework for evolutionary optimization with approximate fitnessfunctions
IEEE Transactions on Evolutionary Computation
Evolutionary optimization in uncertain environments-a survey
IEEE Transactions on Evolutionary Computation
Efficient search for robust solutions by means of evolutionary algorithms and fitness approximation
IEEE Transactions on Evolutionary Computation
Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels
IEEE Transactions on Evolutionary Computation
Proceedings of the 15th annual conference companion on Genetic and evolutionary computation
Annals of Mathematics and Artificial Intelligence
Hi-index | 0.00 |
Meta-modeling has become a crucial tool in solving expensive optimization problems. Much of the work in the past has focused on finding a good regression method to model the fitness function. Examples include classical linear regression, splines, neural networks, Kriging and support vector regression. This paper specifically draws attention to the fact that assessing model accuracy is a crucial aspect in the meta-modeling framework. Resampling strategies such as cross-validation, subsampling, bootstrapping, and nested resampling are prominent methods for model validation and are systematically discussed with respect to possible pitfalls, shortcomings, and specific features. A survey of meta-modeling techniques within evolutionary optimization is provided. In addition, practical examples illustrating some of the pitfalls associated with model selection and performance assessment are presented. Finally, recommendations are given for choosing a model validation technique for a particular setting.