Accuracy of machine learning models versus "hand crafted" expert systems - A credit scoring case study

Authors:
Arie Ben-David;Eibe Frank
Affiliations:
Management Information Systems, Department of Technology Management, Holon Institute of Technology, Holon, Israel;Department of Computer Science, University of Waikato, Hamilton, New Zealand
Venue:
Expert Systems with Applications: An International Journal
Year:
2009

Citing 17
Cited 0

A Nearest Hyperrectangle Learning Method

Machine Learning
Trading MIPS and memory for knowledge engineering

Communications of the ACM
Very Simple Classification Rules Perform Well on Most Commonly Used Datasets

Machine Learning
Bagging predictors

Machine Learning
Locally Weighted Learning

Artificial Intelligence Review - Special issue on lazy learning
On the Optimality of the Simple Bayesian Classifier under Zero-One Loss

Machine Learning - Special issue on learning with probabilistic representations
Fast training of support vector machines using sequential minimal optimization

Advances in kernel methods
Knowledge acquisition by encoding expert rules versus computer induction from examples: a case study involving soybean pathology

International Journal of Human-Computer Studies - Special issue: 1969-1999, the 30th anniversary
MultiBoosting: A Technique for Combining Boosting and Wagging

Machine Learning
A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms

Machine Learning
Random Forests

Machine Learning
Automated Support for Building and Extending Expert Models

Machine Learning
The Power of Decision Tables

ECML '95 Proceedings of the 8th European Conference on Machine Learning
Generating Accurate Rule Sets Without Global Optimization

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
In Defense of One-Vs-All Classification

The Journal of Machine Learning Research
Logistic Model Trees

Machine Learning
Constructing diverse classifier ensembles using artificial training examples

IJCAI'03 Proceedings of the 18th international joint conference on Artificial intelligence

Quantified Score

Hi-index	12.05

Visualization

Abstract

Relatively few publications compare machine learning models with expert systems when applied to the same problem domain. Most publications emphasize those cases where the former beat the latter. Is it a realistic picture of the state of the art? Some other findings are presented here. The accuracy of a real world ''mind crafted'' credit scoring expert system is compared with dozens of machine learning models. The results show that while some machine learning models can surpass the expert system's accuracy with statistical significance, most models do not. More interestingly, this happened only when the problem was treated as regression. In contrast, no machine learning model showed any statistically significant advantage over the expert system's accuracy when the same problem was treated as classification. Since the true nature of the class data was ordinal, the latter is the more appropriate setting. It is also shown that the answer to the question is highly dependent on the meter that is being used to define accuracy.