High-performance sorting on networks of workstations

Authors:
Andrea C. Arpaci-Dusseau;Remzi H. Arpaci-Dusseau;David E. Culler;Joseph M. Hellerstein;David A. Patterson
Affiliations:
Computer Science Division, University of California, Berkeley;-;Computer Science Division, University of California, Berkeley;Computer Science Division, University of California, Berkeley;Computer Science Division, University of California, Berkeley
Venue:
SIGMOD '97 Proceedings of the 1997 ACM SIGMOD international conference on Management of data
Year:
1997

Citing 30
Cited 48

A measure of transaction processing power

Datamation
A benchmark of NonStop SQL on the debit credit transaction

SIGMOD '88 Proceedings of the 1988 ACM SIGMOD international conference on Management of data
Sorting large data files on POOMA

CONPAR 90 Proceedings of the joint international conference on Vector and parallel processing
FastSort: a distributed single-input single-output external sort

SIGMOD '90 Proceedings of the 1990 ACM SIGMOD international conference on Management of data
A comparison of sorting algorithms for the connection machine CM-2

SPAA '91 Proceedings of the third annual ACM symposium on Parallel algorithms and architectures
Radix sort for vector multiprocessors

Proceedings of the 1991 ACM/IEEE conference on Supercomputing
Symmetric multiprocessing in Solaris 2.0

COMPCON '92 Proceedings of the thirty-seventh international conference on COMPCON
Active messages: a mechanism for integrated communication and computation

ISCA '92 Proceedings of the 19th annual international symposium on Computer architecture
Cooperative shared memory: software and hardware for scalable multiprocessors

ACM Transactions on Computer Systems (TOCS)
Parallel programming in Split-C

Proceedings of the 1993 ACM/IEEE conference on Supercomputing
A practical external sort for shared disk MPP's

Proceedings of the 1993 ACM/IEEE conference on Supercomputing
AlphaSort: a RISC machine sort

SIGMOD '94 Proceedings of the 1994 ACM SIGMOD international conference on Management of data
Virtual memory mapped network interface for the SHRIMP multicomputer

ISCA '94 Proceedings of the 21st annual international symposium on Computer architecture
The interaction of parallel and sequential workloads on a network of workstations

Proceedings of the 1995 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems
An overview of DB2 parallel edition

SIGMOD '95 Proceedings of the 1995 ACM SIGMOD international conference on Management of data
Informix online XPS

SIGMOD '95 Proceedings of the 1995 ACM SIGMOD international conference on Management of data
U-Net: a user-level network interface for parallel and distributed computing

SOSP '95 Proceedings of the fifteenth ACM symposium on Operating systems principles
A super scalar sort algorithm for RISC processors

SIGMOD '96 Proceedings of the 1996 ACM SIGMOD international conference on Management of data
Fast Parallel Sorting Under LogP: Experience with the CM-5

IEEE Transactions on Parallel and Distributed Systems
Operating system support for database management

Communications of the ACM
Parallel sorting on a shared-nothing architecture using probabilistic splitting

PDIS '91 Proceedings of the first international conference on Parallel and distributed information systems
Myrinet: A Gigabit-per-Second Local Area Network

IEEE Micro
A Case for NOW (Networks of Workstations)

IEEE Micro
Assessing Fast Network Interfaces

IEEE Micro
Prototyping Bubba, A Highly Parallel Database System

IEEE Transactions on Knowledge and Data Engineering
The Gamma Database Machine Project

IEEE Transactions on Knowledge and Data Engineering
The parameterized Round-Robin partitioned algorithm for parallel external sort

IPPS '95 Proceedings of the 9th International Symposium on Parallel Processing
A memory-adaptive sort (MASORT) for database systems

CASCON '96 Proceedings of the 1996 conference of the Centre for Advanced Studies on Collaborative research
Sorting Large Files on a Backend Multiprocessor

Sorting Large Files on a Backend Multiprocessor
Scalability in the XFS file system

ATEC '96 Proceedings of the 1996 annual conference on USENIX Annual Technical Conference

Effects of communication latency, overhead, and bandwidth in a cluster architecture

Proceedings of the 24th annual international symposium on Computer architecture
Searching for the sorting record: experiences in tuning NOW-Sort

SPDT '98 Proceedings of the SIGMETRICS symposium on Parallel and distributed tools
Active disks: programming model, algorithms and evaluation

Proceedings of the eighth international conference on Architectural support for programming languages and operating systems
Cluster I/O with River: making the fast case common

Proceedings of the sixth workshop on I/O in parallel and distributed systems
Communication conscious radix sort

ICS '99 Proceedings of the 13th international conference on Supercomputing
Parallel sorting on cache-coherent DSM multiprocessors

SC '99 Proceedings of the 1999 ACM/IEEE conference on Supercomputing
Eddies: continuously adaptive query processing

SIGMOD '00 Proceedings of the 2000 ACM SIGMOD international conference on Management of data
Efficient bundle sorting

SODA '00 Proceedings of the eleventh annual ACM-SIAM symposium on Discrete algorithms
Quantitative Characterization and Analysis of the I/O Behavior of a Commercial Distributed-Shared-Memory Machine

IEEE Transactions on Parallel and Distributed Systems
High-Performance Routing in Networks of Workstations with Irregular Topology

IEEE Transactions on Parallel and Distributed Systems
Columnsort lives! an efficient out-of-core sorting program

Proceedings of the thirteenth annual ACM symposium on Parallel algorithms and architectures
Implicit coscheduling: coordinated scheduling with implicit information in distributed systems

ACM Transactions on Computer Systems (TOCS)
Parallel database processing on a 100 Node PC cluster: cases for decision support query processing and data mining

SC '97 Proceedings of the 1997 ACM/IEEE conference on Supercomputing
Run-time adaptation in river

ACM Transactions on Computer Systems (TOCS)
Opportunity Cost Algorithms for Reduction of I/O and Interprocess Communication Overhead in a Computing Cluster

IEEE Transactions on Parallel and Distributed Systems
Scalable Sweeping-Based Spatial Join

VLDB '98 Proceedings of the 24rd International Conference on Very Large Data Bases
Active Storage for Large-Scale Data Mining and Multimedia

VLDB '98 Proceedings of the 24rd International Conference on Very Large Data Bases
DBMSs on a Modern Processor: Where Does Time Go?

VLDB '99 Proceedings of the 25th International Conference on Very Large Data Bases
OLAP Query Processing Algorithm Based on Relational Storage

WAIM '02 Proceedings of the Third International Conference on Advances in Web-Age Information Management
Getting More from Out-of-Core Columnsort

ALENEX '02 Revised Papers from the 4th International Workshop on Algorithm Engineering and Experiments
Parallel R-Tree Search Algorithm on DSVM

DASFAA '99 Proceedings of the Sixth International Conference on Database Systems for Advanced Applications
Exploiting Gray-Box Knowledge of Buffer-Cache Management

ATEC '02 Proceedings of the General Track of the annual conference on USENIX Annual Technical Conference
Bridging the Information Gap in Storage Protocol Stacks

ATEC '02 Proceedings of the General Track of the annual conference on USENIX Annual Technical Conference
Distributed multiprocessor environments

Journal of Computing Sciences in Colleges
Active I/O Switches in System Area Networks

HPCA '03 Proceedings of the 9th International Symposium on High-Performance Computer Architecture
Distributed Computing with Load-Managed Active Storage

HPDC '02 Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing
Out-of-core computing on mesh connected computers

Journal of Parallel and Distributed Computing
A framework for adaptive algorithm selection in STAPL

Proceedings of the tenth ACM SIGPLAN symposium on Principles and practice of parallel programming
Implementing sorting in database systems

ACM Computing Surveys (CSUR)
GPUTeraSort: high performance graphics co-processor sorting for large database management

Proceedings of the 2006 ACM SIGMOD international conference on Management of data
MapReduce: simplified data processing on large clusters

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Millennium sort: a cluster-based application for windows NT using DCOM, river primitives and the virtual interface architecture

WINSYM'99 Proceedings of the 3rd conference on USENIX Windows NT Symposium - Volume 3
MapReduce: simplified data processing on large clusters

Communications of the ACM - 50th anniversary issue: 1958 - 2008
High-speed parallel external sorting of data with arbitrary distribution

International Journal of High Performance Computing and Networking
Parallel external sort of floating-point data by integer conversion

ACC'08 Proceedings of the WSEAS International Conference on Applied Computing Conference
psort, Yet Another Fast Stable Sorting Software

SEA '09 Proceedings of the 8th International Symposium on Experimental Algorithms
Adaptively parallelizing distributed range queries

Proceedings of the VLDB Endowment
The effect of local sort on parallel sorting algorithms

EUROMICRO-PDP'02 Proceedings of the 10th Euromicro conference on Parallel, distributed and network-based processing
Load and storage balanced posting file partitioning for parallel information retrieval

Journal of Systems and Software
psort, yet another fast stable sorting software

Journal of Experimental Algorithmics (JEA)
TritonSort: a balanced large-scale sorting system

Proceedings of the 8th USENIX conference on Networked systems design and implementation
It’s elementary, my dear watson: time-optimal sorting algorithms on a completely overlapping network

ISPA'05 Proceedings of the Third international conference on Parallel and Distributed Processing and Applications
Oblivious vs. distribution-based sorting: an experimental evaluation

ESA'05 Proceedings of the 13th annual European conference on Algorithms
Parallelizing skyline queries for scalable distribution

EDBT'06 Proceedings of the 10th international conference on Advances in Database Technology
Tarazu: optimizing MapReduce on heterogeneous clusters

ASPLOS XVII Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems
CloudRAMSort: fast and efficient large-scale distributed RAM sort on shared-nothing cluster

SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
TritonSort: A Balanced and Energy-Efficient Large-Scale Sorting System

ACM Transactions on Computer Systems (TOCS)
MapReduce with communication overlap (MaRCO)

Journal of Parallel and Distributed Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

We report the performance of NOW-Sort, a collection of sorting implementations on a Network of Workstations (NOW). We find that parallel sorting on a NOW is competitive to sorting on the large-scale SMPs that have traditionally held the performance records. On a 64-node cluster, we sort 6.0 GB in just under one minute, while a 32-node cluster finishes the Datamation benchmark in 2.41 seconds.Our implementations can be applied to a variety of disk, memory, and processor configurations; we highlight salient issues for tuning each component of the system. We evaluate the use of commodity operating systems and hardware for parallel sorting. We find existing OS primitives for memory management and file access adequate. Due to aggregate communication and disk bandwidth requirements, the bottleneck of our system is the workstation I/O bus.