Stargazing through a digital veil: managing a large scale sky survey using distributed databases on HPC clusters

Authors:
Yogesh Simmhan;Catharine van Ingen;Jim Heasley;Alex Szalay
Affiliations:
University of Southern California, Los Angeles, CA, USA;Microsoft Research, San Francisco, CA, USA;University of Hawaii, Honolulu, HI, USA;The Johns Hopkins University , Baltimore, MD, USA
Venue:
Proceedings of the first annual workshop on High performance computing meets databases
Year:
2011

Citing 6
Cited 0

Microsoft TerraServer: a spatial data warehouse

SIGMOD '00 Proceedings of the 2000 ACM SIGMOD international conference on Management of data
Designing and mining multi-terabyte astronomy archives: the Sloan Digital Sky Survey

SIGMOD '00 Proceedings of the 2000 ACM SIGMOD international conference on Management of data
CasJobs and MyDB: A Batch Query Workbench

Computing in Science and Engineering
Validity of the single processor approach to achieving large scale computing capabilities

AFIPS '67 (Spring) Proceedings of the April 18-20, 1967, spring joint computer conference
GrayWulf: Scalable Clustered Architecture for Data Intensive Computing

HICSS '09 Proceedings of the 42nd Hawaii International Conference on System Sciences
Building Reliable Data Pipelines for Managing Community Data Using Scientific Workflows

E-SCIENCE '09 Proceedings of the 2009 Fifth IEEE International Conference on e-Science

Quantified Score

Hi-index	0.01

Visualization

Abstract

The Sloan Digital Sky Survey established the use of relational databases for the scans and cone searches common to astronomy analyses. The Pan-STARRS project scales up SDSS by melding HPC clusters with hierarchical and spatially partitioned distributed databases to meet the challenge of near realtime handling of the multiple data surveys generated by a GigaPixel telescope. This meld provides job management capabilities on the cluster for scientist query submission as well as the backend data updates and fault management necessary for a system with no traditional backup. This paper describes the Pan-STARRS HPC+database experience, highlights the current focus of our work and where further research is needed.