Five-stage procedure for the evaluation of simulation models through statistical techniques
WSC '96 Proceedings of the 28th conference on Winter simulation
Validation of trace-driven simulation models: regression analysis revisited
WSC '96 Proceedings of the 28th conference on Winter simulation
Validation of models: statistical techniques and data availability
Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future - Volume 1
VV&A; IV: validation of trace-driven simulation models: more on bootstrap tests
Proceedings of the 32nd conference on Winter simulation
Proceedings of the 32nd conference on Winter simulation
A simulation tool for combined rail/road transport in intermodal terminals
Mathematics and Computers in Simulation - Selected papers of the MSSANZ/IMACS 13th biennial conference on modelling and simulation, Hamilton, New Zealand, December 1999
Position paper: Characterising performance of environmental models
Environmental Modelling & Software
Environmental Modelling & Software
Hi-index | 0.01 |
This paper argues that it is wrong to require that regressing the outputs of a trace-driven simulation on the observed real outcomes should give a 457 (unit slope) line through the origin (zero intercept). This note proposes instead an alternative requirement: the responses of the simulated and the real systems should have the same means and the same variances. To test statistically whether this requirement is satisfied, a novel procedure is derived: regress the differences between simulated and real responses on their associated sums, and test whether the resulting intercept and slope are both zero. This novel but simple test assumes identically, independently, and normally distributed outputs of the real system and the simulated system. The old and the new procedures are investigated in extensive Monte Carlo experiments that simulate M/M/1 queueing systems. The conclusions are: (i) the naive intuitive test rejects a valid simulation model substantially more often than the novel test does; (ii) the naive test shows "perverse" behavior within a certain domain: the worse the simulation model, the higher its estimated probability of acceptance; and (iii) the novel test does not reject a valid simulation model too often (its type I error probability is correct), provided the queueing response is transformed appropriately to obtain (nearly) normally distributed responses.