Model Monitor (M2): Evaluating, Comparing, and Monitoring Models
The Journal of Machine Learning Research
Estimating accuracy for text classification tasks on large unlabeled data
CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Adaptive methods for classification in arbitrarily imbalanced and drifting data streams
PAKDD'09 Proceedings of the 13th Pacific-Asia international conference on Knowledge discovery and data mining: new frontiers in applied data mining
User generated content consumption and social networking in knowledge-sharing OSNs
SBP'10 Proceedings of the Third international conference on Social Computing, Behavioral Modeling, and Prediction
| Hi-index | 0.00 |
A fundamental tenet assumed by many classification algorithms is the presumption that both training and testing samples are drawn from the same distribution of data this is the stationary distribution assumption. This entails that the past is strongly indicative of the future. However, in real world applications, many factors may alter the One True Model responsible for generating the data distribution both significantly and subtly. In circumstances violating the stationary distribution assumption, traditional validation schemes such as ten-folds and hold-out become poor performance predictors and classifier rankers. Thus, it becomes critical to discover the fracture points in classifier performance by discovering the divergence between populations. In this paper, we implement a comprehensive evaluation framework to identify bias, enabling selection of a "correct" classifier given the sample bias. To thoroughly evaluate the performance of classifiers within biased distributions, we consider the following three scenarios: missing completely at random (akin to stationary); missing at random; and missing not at random. The latter reflects the canonical sample selection bias problem.