The input/output complexity of sorting and related problems
Communications of the ACM
A simple bounded disorder file organization with good performance
ACM Transactions on Database Systems (TODS)
Access methods for multiversion data
SIGMOD '89 Proceedings of the 1989 ACM SIGMOD international conference on Management of data
Introduction to algorithms
The buddy tree: an efficient and robust access method for spatial data base
Proceedings of the sixteenth international conference on Very large databases
The design and implementation of a log-structured file system
ACM Transactions on Computer Systems (TOCS)
The LRU-K page replacement algorithm for database disk buffering
SIGMOD '93 Proceedings of the 1993 ACM SIGMOD international conference on Management of data
RAID: high-performance, reliable secondary storage
ACM Computing Surveys (CSUR)
The log-structured merge-tree (LSM-tree)
Acta Informatica
On-line reorganization of sparsely-populated B+-trees
SIGMOD '96 Proceedings of the 1996 ACM SIGMOD international conference on Management of data
Improving the performance of log-structured file systems with adaptive methods
Proceedings of the sixteenth ACM symposium on Operating systems principles
The art of computer programming, volume 3: (2nd ed.) sorting and searching
The art of computer programming, volume 3: (2nd ed.) sorting and searching
Extendible hashing—a fast access method for dynamic files
ACM Transactions on Database Systems (TODS)
ACM Computing Surveys (CSUR)
Maintaining a large spatial database with T2SM
Proceedings of the 9th ACM international symposium on Advances in geographic information systems
Design of Dynamic Data Structures
Design of Dynamic Data Structures
R-trees: a dynamic index structure for spatial searching
SIGMOD '84 Proceedings of the 1984 ACM SIGMOD international conference on Management of data
Concurrency Control in B-Trees with Batch Updates
IEEE Transactions on Knowledge and Data Engineering
Probabilistic Model and Optimal Reorganization of B+-Tree with Physical Clustering
IEEE Transactions on Knowledge and Data Engineering
STR: A Simple and Efficient Algorithm for R-Tree Packing
ICDE '97 Proceedings of the Thirteenth International Conference on Data Engineering
Incremental Organization for Data Recording and Warehousing
VLDB '97 Proceedings of the 23rd International Conference on Very Large Data Bases
A Generic Approach to Bulk Loading Multidimensional Index Structures
VLDB '97 Proceedings of the 23rd International Conference on Very Large Data Bases
Design, Implementation, and Performance of the LHAM Log-Structured History Data Access Method
VLDB '98 Proceedings of the 24rd International Conference on Very Large Data Bases
On Optimal Node Splitting for R-trees
VLDB '98 Proceedings of the 24rd International Conference on Very Large Data Bases
A Novel Index Supporting High Volume Data Warehouse Insertion
VLDB '99 Proceedings of the 25th International Conference on Very Large Data Bases
The Buffer Tree: A New Technique for Optimal I/O-Algorithms (Extended Abstract)
WADS '95 Proceedings of the 4th International Workshop on Algorithms and Data Structures
Hilbert R-tree: An Improved R-tree using Fractals
VLDB '94 Proceedings of the 20th International Conference on Very Large Data Bases
Generalized Search Trees for Database Systems
VLDB '95 Proceedings of the 21th International Conference on Very Large Data Bases
Rules of Thumb in Data Engineering
ICDE '00 Proceedings of the 16th International Conference on Data Engineering
VLDB '02 Proceedings of the 28th international conference on Very Large Data Bases
Online Random Shuffling of Large Database Tables
IEEE Transactions on Knowledge and Data Engineering
Rose: compressed, log-structured replication
Proceedings of the VLDB Endowment
Story book: an efficient extensible provenance framework
TAPP'09 First workshop on on Theory and practice of provenance
The RDF-3X engine for scalable management of RDF data
The VLDB Journal — The International Journal on Very Large Data Bases
UPI: a primary index for uncertain databases
Proceedings of the VLDB Endowment
bLSM: a general purpose log structured merge tree
SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Hi-index | 0.00 |
The rate of increase in hard disk storage capacity continues to outpace the rate of decrease in hard disk seek time. This trend implies that the value of a seek is increasing exponentially relative to the value of storage.With this trend in mind, we introduce the partitioned exponential file (PE file) which is a generic storage manager that can be customized for many different types of data (e.g., numerical, spatial, or temporal). The PE file is intended for use in environments with intense update loads and concurrent, analytic queries. Such an environment may be found, for example, in long-running scientific applications which can produce petabytes of data. For example, the proposed Large Synoptic Survey Telescope [36] will produce 50---100 petabytes of observational, scientific data over its multi-year lifetime. This database will never be taken off-line, so bursty update loads of tens of terabytes per day must be handled concurrently with data analysis. In the PE file, data are organized as a series of on-disk sorts with a careful, global organization. Because the PE file relies heavily on sequential I/O, only a fraction of a disk seek is required for a typical record insertion or retrieval.In addition to describing the PE file, we also detail a set of benchmarking experiments for T1SM, which is a PE file customized for use with multi-attribute data records ordered on a single numerical attribute. In our benchmarking, we implement and test many competing data organizations that can be used to index and store such data, such as the B+-Tree, the LSM-Tree, the Buffer Tree, the Stepped Merge Method, and the Y-Tree. As expected, no organization is the best over all benchmarks, but our experiments show that T1SM is the best choice in many situations, suggesting that it is the best overall. Specifically, T1SM performs exceptionally well in the case of a heavy query workload that must be handled concurrently with an intense insertion stream. Our experiments show that T1SM (and its close cousin, the T2SM storage manager for spatial data) can handle very heavy mixed workloads of this type, and still maintain acceptably small query latencies.