AlphaSort: a cache-sensitive parallel external sort

  • Authors:
  • Chris Nyberg;Tom Barclay;Zarka Cvetanovic;Jim Gray;Dave Lomet

  • Affiliations:
  • Ordinal Technology Corp., Orinda, CA;Microsoft Corp., Redmond, WA;Digital Equipment Corp., Boxborough, MA;San Francisco, CA;Microsoft Corp., Redmond, WA

  • Venue:
  • The VLDB Journal — The International Journal on Very Large Data Bases
  • Year:
  • 1995

Quantified Score

Hi-index 0.00

Visualization

Abstract

A new sort algorithm, called AlphaSort, demonstrates that commodity processors and disks can handle commercial batch workloads. Using commodity processors, memory, and arrays of SCSI disks, AlphaSort runs the industry-standard sort benchmark in seven seconds. This beats the best published record on a 32-CPU 32-disk Hypercube by 8:1. On another benchmark, AlphaSort sorted more than a gigabyte in one minute. AlphaSort is a cache-sensitive, memory-intensive sort algorithm. We argue that modern architectures require algorithm designers to re-examine their use of the memory hierarchy. AlphaSort uses clustered data structures to get good cache locality, file striping to get high disk bandwidth, QuickSort to generate runs, and replacement-selection to merge the runs. It uses shared memory multiprocessors to break the sort into subsort chores. Because startup times are becoming a significant part of the total time, we propose two new benchmarks: (1) MinuteSort: how much can you sort in one minute, and (2) PennySort: how much can you sort for one penny.