Assessing the impact of changing environments on classifier performance

Authors:
Rocío Alaiz-Rodríguez;Nathalie Japkowicz
Affiliations:
Dpto. de Ingeniería Eléctrica y de Sistemas y Automática, Universidad de León, León, Spain;SITE, University of Ottawa., Ottawa, Ontario, Canada
Venue:
Canadian AI'08 Proceedings of the Canadian Society for computational studies of intelligence, 21st conference on Advances in artificial intelligence
Year:
2008

Citing 7
Cited 8

The impact of changing populations on classifier performance

KDD '99 Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining
Data mining: practical machine learning tools and techniques with Java implementations

Data mining: practical machine learning tools and techniques with Java implementations
Robust Classification for Imprecise Environments

Machine Learning
Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure

Neural Computation
Cost curves: An improved method for visualizing classifier performance

Machine Learning
Minimax Regret Classifier for Imprecise Class Distributions

The Journal of Machine Learning Research
Asymptotic Bayesian generalization error when training and test distributions are different

Proceedings of the 24th international conference on Machine learning

Quantification and semi-supervised classification methods for handling changes in class distribution

Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
A unifying view on dataset shift in classification

Pattern Recognition
Analysis of preprocessing vs. cost-sensitive learning for imbalanced classification. Open problems on intrinsic data characteristics

Expert Systems with Applications: An International Journal
Robustness of classifiers to changing environments

AI'10 Proceedings of the 23rd Canadian conference on Advances in Artificial Intelligence
Handling concept drift via ensemble and class distribution estimation technique

ADMA'11 Proceedings of the 7th international conference on Advanced Data Mining and Applications - Volume Part II
Online linear and quadratic discriminant analysis with adaptive forgetting for streaming classification

Statistical Analysis and Data Mining
Class distribution estimation based on the Hellinger distance

Information Sciences: an International Journal
On the importance of the validation technique for classification with imbalanced datasets: Addressing covariate shift when data is skewed

Information Sciences: an International Journal

Quantified Score

Hi-index	0.00

Visualization

Abstract

The purpose of this paper is to test the hypothesis that simple classifiers are more robust to changing environments than complex ones. We propose a strategy for generating artificial, but realistic domains, which allows us to control the changing environment and test a variety of situations. Our results suggest that evaluating classifiers on such tasks is not straightforward since the changed environment can yield a simpler or more complex domain. We propose a metric capable of taking this issue into consideration and evaluate our classifiers using it. We conclude that in mild cases of population drifts simple classifiers deteriorate more than complex ones and that in more severe cases as well as in class definition changes, all classifiers deteriorate to about the same extent. This means that in all cases, complex classifiers remain more accurate than simpler ones, thus challenging the hypothesis that simple classifiers are more robust to changing environments than complex ones.