A distributed multi-storage I/O system for data intensive scientific computing

  • Authors:

  • Xiaohui Shen;Alok Choudhary

  • Affiliations:

  • Department of Electrical and Computer Engineering, Center for Parallel and Distributed Computing, Northwestern University, Evanston, IL;Department of Electrical and Computer Engineering, Center for Parallel and Distributed Computing, Northwestern University, Evanston, IL

  • Venue:
  • Parallel Computing - Special issue: Parallel and distributed scientific and engineering computing
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

More and more parallel applications are running in a distributed environment to take advantage of easily available and inexpensive commodity resources. For data intensive applications, employing multiple distributed storage resources has many advantages. In this paper, we present a Multi-Storage I/O System (MS-I/O) that cannot only effectively manage various distributed storage resources in the system, but also provide novel high performance storage access schemes. MS-I/O employs many state-of-the-art I/O optimizations such as collective I/O, asynchronous I/O etc. and a number of new techniques such as data location, data replication, subtile, superfile and data access history. In addition, many MS-I/O optimization schemes can work simultaneously within a single data access session, greatly improving the performance.Although I/O optimization techniques can help improve performance, it also complicates I/ O system. In addition, most optimization techniques have their limitations. Therefore, selecting accurate optimization policies requires expert knowledge which is not suitable for end users who may have little knowledge of I/O techniques. So the task of I/O optimization decision should be left to the I/O system itself, that is, automatic from user's point of view. We present a User Access Pattern data structure which is associated with each dataset that can help MS-I/O easily make accurate I/O optimization decisions.