Statistics in simulation: How to design for selecting the best alternative

• Authors: Edward J. Dudewicz
• Venue: WSC '76 Proceedings of the 76 Bicentennial conference on Winter simulation
• Year: 1976

In many simulation studies the experimenter (the person running the simulation) has under consideration several (two or more) proposed procedures (e.g., for running a real-world system), and is simulating in order to determine which is the best procedure (with regard to certain specified criteria of “goodness”). Such an experimenter does not wish basically to test hypotheses, or construct confidence intervals, or perform regression analyses (though these may be appropriate minor parts of his analysis); he does wish basically to select the best of several procedures, and the major part of his analysis should therefore be directed towards this goal. It is precisely for this problem that ranking-and-selection procedures were developed. These procedures set sample size (in simulation this means run-length) explicitly so as to guarantee that the probability that “the procedure actually selected by the experimenter is the best procedure” is suitably large. In this paper we first review the background ideas of ranking-and-selection and contrast them to other approaches to multi-population problems (which, while sometimes appropriate in such areas as social science experimentation, are almost wholly inappropriate for use in statistical design and analysis of simulation experiments). Recommended procedures for several common situations are then outlined in detail. References where further theoretical details may be obtained are provided, along with information on current developments in the area. It is intended that the motivation and technical detail given be sufficient for intelligent application in many common situations (though other situations will still require supplementary consultation).