ACIC: automatic cloud I/O configurator for HPC applications

  • Authors:

  • Mingliang Liu;Ye Jin;Jidong Zhai;Yan Zhai;Qianqian Shi;Xiaosong Ma;Wenguang Chen

  • Affiliations:

  • Tsinghua University and Tsinghua University in Shenzhen;North Carolina State University;Tsinghua University;University of Wisconsin-Madison;North Carolina State University and Oak Ridge National Laboratory;North Carolina State University and Oak Ridge National Laboratory;Tsinghua University and Tsinghua University in Shenzhen

  • Venue:
  • SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

The cloud has become a promising alternative to traditional HPC centers or in-house clusters. This new environment highlights the I/O bottleneck problem, typically with top-of-the-line compute instances but sub-par communication and I/O facilities. It has been observed that changing cloud I/O system configurations leads to significant variation in the performance and cost efficiency of I/O intensive HPC applications. However, storage system configuration is tedious and error-prone to do manually, even for experts. This paper proposes ACIC, which takes a given application running on a given cloud platform, and automatically searches for optimized I/O system configurations. ACIC utilizes machine learning models to perform black-box performance/cost predictions. To tackle the high-dimensional parameter exploration space unique to cloud platforms, we enable affordable, reusable, and incremental training guided by Plackett and Burman Matrices. Results with four representative applications indicate that ACIC consistently identifies near-optimal configurations among a large group of candidate settings.