Design and analysis of a multi-dimensional data sampling service for large scale data analysis applications

Authors:
Xi Zhang;Tahsin Kurc;Joel Saltz;Srinivasan Parthasarathy
Affiliations:
Department of Biomedical Informatics and Department of Computer Science and Engineering, The Ohio State University, Columbus, OH;Department of Biomedical Informatics, The Ohio State University, Columbus, OH;Department of Biomedical Informatics and Department of Computer Science and Engineering, The Ohio State University, Columbus, OH;Department of Biomedical Informatics and Department of Computer Science and Engineering, The Ohio State University, Columbus, OH
Venue:
IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing
Year:
2006

Citing 17
Cited 1

Random sampling with a reservoir

ACM Transactions on Mathematical Software (TOMS)
Linear clustering of objects with multiple attributes

SIGMOD '90 Proceedings of the 1990 ACM SIGMOD international conference on Management of data
Parallel database systems: the future of high performance database systems

Communications of the ACM
Study of scalable declustering algorithms for parallel grid files

Study of scalable declustering algorithms for parallel grid files
Efficient progressive sampling

KDD '99 Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining
Disk allocation for Cartesian product files on multiple-disk systems

ACM Transactions on Database Systems (TODS)
Querying multi-dimensional data indexed using the Hilbert space-filling curve

ACM SIGMOD Record
Analysis of the Clustering Properties of the Hilbert Space-Filling Curve

IEEE Transactions on Knowledge and Data Engineering
Sampling from Spatial Databases

Proceedings of the Ninth International Conference on Data Engineering
A General Method for Scaling Up Machine Learning Algorithms and its Application to Clustering

ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Sampling Large Databases for Association Rules

VLDB '96 Proceedings of the 22th International Conference on Very Large Data Bases
Random Sampling from Database Files: A Survey

Proceedings of the 5th International Conference SSDBM on Statistical and Scientific Database Management
A Similarity Graph-Based Approach to Declustering Problems and Its Application towards Paralleling Grid Files

ICDE '95 Proceedings of the Eleventh International Conference on Data Engineering
Efficient Progressive Sampling for Association Rules

ICDM '02 Proceedings of the 2002 IEEE International Conference on Data Mining
Online maintenance of very large random samples

SIGMOD '04 Proceedings of the 2004 ACM SIGMOD international conference on Management of data
Design of a next generation sampling service for large scale data analysis applications

Proceedings of the 19th annual international conference on Supercomputing
Towards dynamically adaptive weather analysis and forecasting in LEAD

ICCS'05 Proceedings of the 5th international conference on Computational Science - Volume Part II

I/O conscious algorithm design and systems support for data analysis on emerging architectures

IPDPS'06 Proceedings of the 20th international conference on Parallel and distributed processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Sampling is a widely used technique to increase efficiency in database and data mining applications operating on large dataset. In this paper we present a scalable sampling implementation that supports efficient, multi-dimensional spatio-temporal sample generation on dynamic, large scale datasets stored on a storage cluster. The proposed algorithm leverages Hilbert space-filling curves in order to provide an approximate linear order of multidimensional data while maintaining spatial locality. This new implementation is then bootstrapped on top of our previous implementation, which efficiently samples large datasets along a single dimension (e.g., time), thereby realizing a service for spatio-temporal sampling. We evaluate the performance of our approach comparing it to the popular R-tree based technique. The experimental results show that our approach achieves up to an order of magnitude higher efficiency and scalability.