Multi-class ROC analysis from a multi-objective optimisation perspective
Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
EMO '09 Proceedings of the 5th International Conference on Evolutionary Multi-Criterion Optimization
A Bayesian Methodology for Estimating Uncertainty of Decisions in Safety-Critical Systems
Proceedings of the 2006 conference on Integrated Intelligent Systems for Engineering Design
The relationship between search based software engineering and predictive modeling
Proceedings of the 6th International Conference on Predictive Models in Software Engineering
A study of the bi-objective next release problem
Empirical Software Engineering
Search-based software engineering: Trends, techniques and applications
ACM Computing Surveys (CSUR)
| Hi-index | 0.00 |
Many safety related and critical systems warn of potentially dangerous events; for example, the short term conflict alert (STCA) system warns of airspace infractions between aircraft. Although installed with current technology, such critical systems may become out of date due to changes in the circumstances in which they function, operational procedures, and the regulatory environment. Current practice is to "tune," by hand, the many parameters governing the system in order to optimize the operating point in terms of the true positive and false positive rates, which are frequently associated with highly imbalanced costs. We cast the tuning of critical systems as a multiobjective optimization problem. We show how a region of the optimal receiver operating characteristic (ROC) curve may be obtained, permitting the system operators to select the operating point. We apply this methodology to the STCA system, using a multiobjective (1+1) evolution strategy, showing that we can improve upon the current hand-tuned operating point, as well as providing the salient ROC curve describing the true positive versus false positive tradeoff. We also provide results for three-objective optimization of the alert response time in addition to the true and false positive rates. Additionally, we illustrate the use of bootstrapping for representing evaluation uncertainty on estimated Pareto fronts, where the evaluation of a system is based upon a finite set of representative data.