The end of an architectural era: (it's time for a complete rewrite)
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
OLTP through the looking glass, and what we found there
Proceedings of the 2008 ACM SIGMOD international conference on Management of data
H-store: a high-performance, distributed main memory transaction processing system
Proceedings of the VLDB Endowment
HYRISE: a main memory hybrid storage engine
Proceedings of the VLDB Endowment
ARC: a self-tuning, low overhead replacement cache
FAST'03 Proceedings of the 2nd USENIX conference on File and storage technologies
CAR: clock with adaptive replacement
FAST'04 Proceedings of the 3rd USENIX conference on File and storage technologies
HyPer: A hybrid OLTP&OLAP main memory database system based on virtual memory snapshots
ICDE '11 Proceedings of the 2011 IEEE 27th International Conference on Data Engineering
SAP HANA database: data management for modern business applications
ACM SIGMOD Record
High-performance concurrency control mechanisms for main-memory databases
Proceedings of the VLDB Endowment
Compacting transactional data in hybrid OLTP&OLAP databases
Proceedings of the VLDB Endowment
Anti-caching: a new approach to database management system architecture
Proceedings of the VLDB Endowment
Hi-index | 0.00 |
Even though main memory is becoming large enough to fit most OLTP databases, it may not always be the best option. OLTP workloads typically exhibit skewed access patterns where some records are hot (frequently accessed) but many records are cold (infrequently or never accessed). Therefore, it is more economical to store the coldest records on a fast secondary storage device such as a solid-state disk. However, main-memory DBMS have no knowledge of secondary storage, while traditional disk-based databases, designed for workloads where data resides on HDD, introduce too much overhead for the common case where the working set is memory resident. In this paper, we propose a simple and low-overhead technique that enables main-memory databases to efficiently migrate cold data to secondary storage by relying on the OS's virtual memory paging mechanism. We propose to log accesses at the tuple level, process the access traces offline to identify relevant access patterns, and then transparently re-organize the in-memory data structures to reduce paging I/O and improve hit rates. The hot/cold data separation is performed on demand and incrementally through careful memory management, without any change to the underlying data structures. We validate experimentally the data re-organization proposal and show that OS paging can be efficient: a TPC-C database can grow two orders of magnitude larger than the available memory size without a noticeable impact on performance.