Integrating parallel file I/O and database support for high-performance scientific data management
Proceedings of the 2000 ACM/IEEE conference on Supercomputing
A performance comparison of bitmap indexes
Proceedings of the tenth international conference on Information and knowledge management
ADC '01 Proceedings of the 12th Australasian database conference
Strategies for processing ad hoc queries on large data warehouses
Proceedings of the 5th ACM international workshop on Data Warehousing and OLAP
A retrieval technique for high-dimensional data and partially specified queries
Data & Knowledge Engineering
Clustering High Dimensional Massive Scientific Datasets
Journal of Intelligent Information Systems
A Scientific Data Management System for Irregular Applications
IPDPS '01 Proceedings of the 15th International Parallel & Distributed Processing Symposium
Optimizing Queries on Compressed Bitmaps
VLDB '00 Proceedings of the 26th International Conference on Very Large Data Bases
Shared Index Scans for Data Warehouses
DaWaK '01 Proceedings of the Third International Conference on Data Warehousing and Knowledge Discovery
Improving the Performance of High-Energy Physics Analysis through Bitmap Indices
DEXA '00 Proceedings of the 11th International Conference on Database and Expert Systems Applications
Bitmap Indices for Speeding Up High-Dimensional Data Analysis
DEXA '02 Proceedings of the 13th International Conference on Database and Expert Systems Applications
Science: knowledge discovery in high-energy particle and nuclear physics
Handbook of data mining and knowledge discovery
Coordinating Simultaneous Caching of File Bundles from Tertiary Storage
SSDBM '00 Proceedings of the 12th International Conference on Scientific and Statistical Database Management
High-performance scientific data management system
Journal of Parallel and Distributed Computing
Multidimensionality in statistical, OLAP, and scientific databases
Multidimensional databases
Journal of Systems and Software - Special issue: Performance modeling and analysis of computer systems and networks
Compressing Bitmap Indices by Data Reorganization
ICDE '05 Proceedings of the 21st International Conference on Data Engineering
Optimizing bitmap indices with efficient compression
ACM Transactions on Database Systems (TODS)
Approximate encoding for direct access and query processing over compressed bitmaps
VLDB '06 Proceedings of the 32nd international conference on Very large data bases
Processing partially specified queries over high-dimensional databases
Data & Knowledge Engineering
On the performance of bitmap indices for high cardinality attributes
VLDB '04 Proceedings of the Thirtieth international conference on Very large data bases - Volume 30
Cooperative scans: dynamic bandwidth sharing in a DBMS
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
Breaking the Curse of Cardinality on Bitmap Indexes
SSDBM '08 Proceedings of the 20th international conference on Scientific and Statistical Database Management
Hi-index | 0.00 |
In many scientific domains, experimental devices or simulation programs generate large volumes of data. The volumes of data may reach hundreds of terabytes and therefore it is impractical to store them on disk systems. Rather they are stored on robotic tape systems that are managed by some mass storage system (MSS). A major bottleneck in analyzing the simulated/collected data is the retrieval of subsets from the tertiary storage system. In this paper we describe the architecture and implementation of a Storage Access Coordination System (STACS) designed to optimize the use of a disk cache, and thus minimize the number of files read from tape. We achieve this by using a specialized index to locate the relevant data on tapes, and by coordinating file caching over multiple queries.We focus on a specific application area, a high energy physics data management and analysis environment. STACS was implemented and is being incorporated in an operational system, scheduled to go on-line in the end of 1999. We also include the results of various tests that demonstrate the benefits and efficiency gained of using the STACS.