Integrating parallel file I/O and database support for high-performance scientific data management

  • Authors:

  • Jaechun No;Rajeev Thakur;Alok Choudhary

  • Affiliations:

  • Math. and Computer Science Division, Argonne National Laboratory, Argonne, IL;Math. and Computer Science Division, Argonne National Laboratory, Argonne, IL;Dept. of Electrical and Computer Eng., Northwestern University, Evanston, IL

  • Venue:
  • Proceedings of the 2000 ACM/IEEE conference on Supercomputing
  • Year:
  • 2000

Quantified Score

Hi-index 0.00

Visualization

Abstract

Many scientific applications have large I/O requirements, in terms of both the size of data and the number of files or data sets. Management, storage, efficient access, and analysis of these data present an extremely challenging task. Traditionally, two different solutions are used for this problem: file I/O or databases. File I/O can provide high performance but is tedious to use with large numbers of files and large and complex data sets. Databases can be convenient, flexible, and powerful but do notperform and scale well for parallel supercomputing applications. We have developed a software system, called Scientific Data Manager (SDM), which aims to combine the good features of both file I/O and databases. SDM provides a high-level API to the user and, internally, uses a parallel file system to store real data and a database to store appreciation-related metadata. SDM takes advantage of various I/O optimizations available in MPI-IO, such as collective I/O and noncontiguous requests, in a manner that is transparent to the user. As a result, users can write and retrieve data with the performance of parallel file I/O, without having to bother with the details of actually performing file I/O.In this paper, we describe the design and implementation of SDM. With the help of two parallel application templates, ASTRO3D and an Euler solver, we illustrate how some of the design criteria affect performance.