On estimating probabilities in tree pruning
EWSL-91 Proceedings of the European working session on learning on Machine learning
Numerical recipes in C (2nd ed.): the art of scientific computing
Numerical recipes in C (2nd ed.): the art of scientific computing
An introduction to Kolmogorov complexity and its applications
An introduction to Kolmogorov complexity and its applications
The nature of statistical learning theory
The nature of statistical learning theory
Machine Learning
Algorithmic stability and sanity-check bounds for leave-one-out cross-validation
COLT '97 Proceedings of the tenth annual conference on Computational learning theory
Combining labeled and unlabeled data with co-training
COLT' 98 Proceedings of the eleventh annual conference on Computational learning theory
Reliable Classifications with Machine Learning
ECML '02 Proceedings of the 13th European Conference on Machine Learning
Ridge Regression Confidence Machine
ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Improving Regressors using Boosting Techniques
ICML '97 Proceedings of the Fourteenth International Conference on Machine Learning
Transduction with Confidence and Credibility
IJCAI '99 Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence
A Brief Introduction to Boosting
IJCAI '99 Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence
Data perturbation for escaping local maxima in learning
Eighteenth national conference on Artificial intelligence
The Journal of Machine Learning Research
Introduction to Machine Learning (Adaptive Computation and Machine Learning)
Introduction to Machine Learning (Adaptive Computation and Machine Learning)
LIBSVM: A library for support vector machines
ACM Transactions on Intelligent Systems and Technology (TIST)
UAI'98 Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence
Empirical Analysis of Reliability Estimates for Individual Regression Predictions
DaWaK '08 Proceedings of the 10th international conference on Data Warehousing and Knowledge Discovery
Comparison of approaches for estimating reliability of individual regression predictions
Data & Knowledge Engineering
An overview of advances in reliability estimation of individual predictions in machine learning
Intelligent Data Analysis
The Knowledge Engineering Review
Expert Systems with Applications: An International Journal
Surrogate modeling in the evolutionary optimization of catalytic materials
Proceedings of the 14th annual conference on Genetic and evolutionary computation
Robust re-identification using randomness and statistical learning: Quo vadis
Pattern Recognition Letters
| Hi-index | 0.00 |
For a given prediction model, some predictions may be reliable while others may be unreliable. The average accuracy of the system cannot provide the reliability estimate for a single particular prediction. The measure of individual prediction reliability can be important information in risk-sensitive applications of machine learning (e.g. medicine, engineering, business). We define empirical measures for estimation of prediction accuracy in regression. Presented measures are based on sensitivity analysis of regression models. They estimate reliability for each individual regression prediction in contrast to the average prediction reliability of the given regression model. We study the empirical sensitivity properties of five regression models (linear regression, locally weighted regression, regression trees, neural networks, and support vector machines) and the relation between reliability measures and distribution of learning examples with prediction errors for all five regression models. We show that the suggested methodology is appropriate only for the three studied models: regression trees, neural networks, and support vector machines, and test the proposed estimates with these three models. The results of our experiments on 48 data sets indicate significant correlations of the proposed measures with the prediction error.