Implementing a general spatial indexing library for relational databases of large numerical simulations

  • Authors:
  • Gerard Lemson;Tamás Budavári;Alexander Szalay

  • Affiliations:
  • MPA;JHU;JHU

  • Venue:
  • SSDBM'11 Proceedings of the 23rd international conference on Scientific and statistical database management
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Large multi-terabyte numerical simulations of different physical systems consist of billions of particles or grid points and hundreds to thousands of snapshots. Increasingly these data sets are stored in large object-relational databases. Most statistical analyses involve extracting various spatio-temporal subsets. Existing built-in spatial indexes in commercial systems lack essential features required for many applications in the physical sciences. We describe a library that we have implemented in several languages and platforms (Java/Oracle, C#/SQL Server) based on generic space-filling curves, implemented as plug-ins. The index provides a mapping of higher dimensional space into the standard linear B-tree index of any relational database. The architecture allows intersections with different geometric primitives. The library has been used for cosmological N-body simulations and isotropic turbulence, providing sub-second response time over datasets exceeding several tens of terabytes. The library can also address complex space-time challenges, like temporal look-back into past light-cones of cosmological simulations.