On the evaluation of simulation results in complex and value-critical decision environments by statistical measures

  • Authors:
  • Marko A. Hofmann

  • Affiliations:
  • University of the Federal Armed Forces Munich, Germany, Neubiberg

  • Venue:
  • Proceedings of the 2008 Summer Computer Simulation Conference
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

Military decision making is generally a paramount example for a complex and value-critical decision environment (which is often even time-critical). Using simulations in order to support decision makers in such environments demands a self-critical reflection of the simulation results, since no single simulation run can be of much predictive validity (at least ex ante). Therefore, one tries to capture the complexity and uncertainty of the "real world" with stochastic simulation models. Their evaluation is based on the statistical analysis of a sequence of N simulation runs. This standard approach leads to a sample mean and variance which are subsequently the central basis for every advice based on the simulation model. There are several critical questions about this approach, some of them are epistemological, like, for example: It is really sufficient to capture the uncertainty of a complex real world system by stochastic variation? Other questions are more pragmatic: What is a sensible measure of effectiveness for the use of such aggregated simulation results in military practice. The acid test of such an approach will always be prediction, regardless of the importance of the slogan "insight not numbers". Hence, we must ask, whether these results help to reduce the rate of error in forecasting or not? And if they do, to what extent? In statistics such questions are answered by so called PRE-measures (proportional reduction in error). The paper -- which presents work in progress -- examines how such measures can be constructed and used for simulation supported military decision making. Taking epistemological consideration into account it demonstrates the possibilities and (narrow) limits of this approach. The basic intention of the paper is to raise further discussions on the topic.