Data Abstraction through Density Estimation by Storage Management

• Authors: Kathrin Anne Meier
• Venue: SSDBM '97 Proceedings of the Ninth International Conference on Scientific and Statistical Database Management
• Year: 1997

One way to cope with the constantly growing amount of scientific data to be analyzed is to derive data abstractions from the original data. Data abstractions can provide a representation of the data in compressed form where the data's semantic structure is maintained. We have explored data abstractions based on} density estimation. Our method to estimate the density of scientific data sets is based on the directory of a multidimensional data access structure. This data density estimator is called directory estimator. It is based on multidimensional adaptive histograms and is therefore computationally efficient, even for large data sets and many dimensions.This paper describes the methodology in general and focuses on the estimator's accuracy in particular. The accuracy of the directory estimator depends on the parameters of the access structures used, such as the bucket capacity. We evaluate the choice of bucket capacity theoretically as well as empirically with the ISE (Integrated Squared Error) being the measure of error and using a gridfile as the data access structure.A useful application of the directory estimator in the field of scientific data is presented with a practical example from astronomy.