The perils of using simulations to evaluate Massively Multiplayer Online Game performance

  • Authors:
  • Alexandre Denault;Jörg Kienzle

  • Affiliations:
  • McGill University, Montreal, QC, Canada;McGill University, Montreal, QC, Canada

  • Venue:
  • Proceedings of the 3rd International ICST Conference on Simulation Tools and Techniques
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

To test and benchmark Massively Multiplayer Online Games (MMOGs) requires hundreds, if not thousands of human players. Given this impracticality, many researchers substitute experimentation with simulation. However, when investigating performance and scalability issues, simulated experiments often yield results that heavily depend on the experimental setup. This paper critically reflects on the use of simulation to conduct experiments in MMOGs. Using Mammoth, a MMOGs research framework, performance measurements such as CPU usage, memory usage and used network bandwidth are collected while running the same game scenario using five different simulation setups. The results are analyzed, and the discovered differences are discussed. The paper concludes that experiments which are aimed at measuring the performance of a MMOGs using simulations must be designed very carefully, especially if they are run on a single machine.