A confidence paradigm for classification systems with out-of-library considerations

Authors:
Nathan J. Leap;Kenneth W. Bauer
Affiliations:
Air Force Institute of Technology, ENS, Wright-Patterson AFB, OH;Air Force Institute of Technology, ENS, Wright-Patterson AFB, OH
Venue:
Intelligent Decision Technologies
Year:
2012

Citing 9
Cited 0

Confidence estimation for speculation control

Proceedings of the 25th annual international symposium on Computer architecture
Strategic Decision Making

Strategic Decision Making
Neural - Network Based Measures of Confidence for Word Recognition

ICASSP '97 Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97)-Volume 2 - Volume 2
Pattern Classification (2nd Edition)

Pattern Classification (2nd Edition)
Using Informational Confidence Values for Classifier Combination: An Experiment with Combined On-Line/Off-Line Japanese Character Recognition

IWFHR '04 Proceedings of the Ninth International Workshop on Frontiers in Handwriting Recognition
Pruning dynamic slices with confidence

Proceedings of the 2006 ACM SIGPLAN conference on Programming language design and implementation
Representing classifier confidence in the safety critical domain: an illustration from mortality prediction in trauma cases

Neural Computing and Applications
A mathematical framework to optimize ATR systems with non-declarations and sensor fusion

Computers and Operations Research
Out-of-Domain Utterance Detection Using Classification Confidences of Multiple Topics

IEEE Transactions on Audio, Speech, and Language Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

There is no universally accepted methodology to determine how much confidence one should place in the output of a classification system. Leap and Bauer [15] present a confidence paradigm based on the assumptions that system confidence acts like, or can be modeled as value and that indication confidence can be modeled as a function of the posterior probability estimates. This paper extends the paradigm to include out-of-library considerations. In addition, a novel out-of-library detector is presented. Developing the out-of-library detector involves bounding and discretizing the feature space and assigning each discrete point to either in-library or out-of-library classes based upon Mahalanobis distance from the in-library target classes. Application of the confidence paradigm to the out-of-library detector leads us to the demonstration of a new concept called out-of-library non-declarations. The extended paradigm is applied to a synthetic data set as well as an automatic target recognition data set. In all cases, the results show performance that tracks well with previous studies found in the literature and demonstrate positive steps toward fuller development of a theoretical framework that unites the viewpoints of the classification system developer and its user.