A technique for moving large data sets over high-performance long distance networks

Authors:
Bradley W. Settlemyer;Jonathan D. Dobson;Stephen W. Hodson;Jeffery A. Kuehn;Stephen W. Poole;Thomas M. Ruwart
Affiliations:
Oak Ridge National Laboratory, One Bethel Valley Road, P.O. Box 2008, 37831-6164, USA;Oak Ridge National Laboratory, One Bethel Valley Road, P.O. Box 2008, 37831-6164, USA;Oak Ridge National Laboratory, One Bethel Valley Road, P.O. Box 2008, 37831-6164, USA;Oak Ridge National Laboratory, One Bethel Valley Road, P.O. Box 2008, 37831-6164, USA;Oak Ridge National Laboratory, One Bethel Valley Road, P.O. Box 2008, 37831-6164, USA;Oak Ridge National Laboratory, One Bethel Valley Road, P.O. Box 2008, 37831-6164, USA
Venue:
MSST '11 Proceedings of the 2011 IEEE 27th Symposium on Mass Storage Systems and Technologies
Year:
2011

Citing 0
Cited 1

Asynchronous object storage with QoS for scientific and commercial big data

PDSW '13 Proceedings of the 8th Parallel Data Storage Workshop

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper we look at the performance characteristics of three tools used to move large data sets over dedicated long distance networking infrastructure. Although performance studies of wide area networks have been a frequent topic of interest, performance analyses have tended to focus on network latency characteristics and peak throughput using network traffic generators. In this study we instead perform an end-to-end long distance networking analysis that includes reading large data sets from a source file system and committing the data to a remote destination file system. An evaluation of end-to-end data movement is also an evaluation of the system configurations employed and the tools used to move the data. For this paper, we have built several storage platforms and connected them with a high performance long distance network configuration. We use these systems to analyze the capabilities of three data movement tools: BBcp, GridFTP, and XDD. Our studies demonstrate that existing data movement tools do not provide efficient performance levels or exercise the storage devices in their highest performance modes.