Pitfalls for testbed evaluations of internet systems

  • Authors:
  • David R. Choffnes;Fabian E. Bustamante

  • Affiliations:
  • Northwestern University, Evanston, IL, USA;Northwestern University, Evanston, IL, USA

  • Venue:
  • ACM SIGCOMM Computer Communication Review
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

Today's open platforms for network measurement and distributed system research, which we collectively refer to as testbeds in this article, provide opportunities for controllable experimentation and evaluations of systems at the scale of hundreds or thousands of hosts. In this article, we identify several issues with extending results from such platforms to Internet wide perspectives. Specifically, we try to quantify the level of inaccuracy and incompleteness of testbed results when applied to the context of a large-scale peer-to-peer (P2P) system. Based on our results, we emphasize the importance of measurements in the appropriate environment when evaluating Internet-scale systems.