The record-breaking terabyte sort on a compaq cluster

  • Authors:
  • Samuel A. Fineberg;Pankaj Mehra

  • Affiliations:
  • Compaq Tandem Labs, Vallco Parkway, Cupertino, CA;Compaq Tandem Labs, Vallco Parkway, Cupertino, CA

  • Venue:
  • WINSYM'99 Proceedings of the 3rd conference on USENIX Windows NT Symposium - Volume 3
  • Year:
  • 1999

Quantified Score

Hi-index 0.01

Visualization

Abstract

Sandia National Laboratories (U.S. Department of Energy) and Compaq Computer Corporation built a 72-node Windows NT cluster, which Sandia utilizes for production work contracted by the U.S. government. Recently, Sandia and Compaq's Tandem Division collaborated on a project to run a 1-terabyte commercial-quality scalable sort on this cluster. The audited result was a new world record of 46.9 minutes, three times faster than the previous record held by a 32-processor shared-memory UNIX system. The external sort utilizes a unique, scalable algorithm that allows near-linear cluster scalability. The sort application exploits several key hardware and software technologies; these include dense-racking Pentium II based servers, Windows NT Workstation 4.0, the Virtual Interface Architecture, and the ServerNet I System Area Network (SAN). The sort code was highly CPU-efficient and stressed asynchronous and sequential I/O and IPC performance. The I/O performance when combined with a high-performance SAN, yielded supercomputer-class performance.