ACM Computing Surveys (CSUR)
The datacycle architecture for very high throughput database systems
SIGMOD '87 Proceedings of the 1987 ACM SIGMOD international conference on Management of data
Parallel database systems: the future of high performance database systems
Communications of the ACM
Communications of the ACM - Special issue on information filtering
Broadcast disks: data management for asymmetric communication environments
SIGMOD '95 Proceedings of the 1995 ACM SIGMOD international conference on Management of data
Balancing push and pull for data broadcast
SIGMOD '97 Proceedings of the 1997 ACM SIGMOD international conference on Management of data
Towards self-tuning data placement in parallel database systems
SIGMOD '00 Proceedings of the 2000 ACM SIGMOD international conference on Management of data
The state of the art in distributed query processing
ACM Computing Surveys (CSUR)
Chord: A scalable peer-to-peer lookup service for internet applications
Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications
A scalable content-addressable network
Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications
The SDSS skyserver: public access to the sloan digital sky server data
Proceedings of the 2002 ACM SIGMOD international conference on Management of data
Optimizing Main-Memory Join on Modern Hardware
IEEE Transactions on Knowledge and Data Engineering
Data Staging for On-Demand Broadcast
Proceedings of the 27th International Conference on Very Large Data Bases
GPUTeraSort: high performance graphics co-processor sorting for large database management
Proceedings of the 2006 ACM SIGMOD international conference on Management of data
ISPASS '03 Proceedings of the 2003 IEEE International Symposium on Performance Analysis of Systems and Software
A high-performance computing method for data allocation in distributed database systems
The Journal of Supercomputing
Query processing methods considering the deadline of queries for database broadcasting systems
Systems and Computers in Japan
MonetDB/SQL Meets SkyServer: the Challenges of a Scientific Database
SSDBM '07 Proceedings of the 19th International Conference on Scientific and Statistical Database Management
Allocating Resources to Parallel Query Plans in Data Grids
GCC '07 Proceedings of the Sixth International Conference on Grid and Cooperative Computing
DB2 design advisor: integrated automatic physical database design
VLDB '04 Proceedings of the Thirtieth international conference on Very large data bases - Volume 30
Self-tuning database systems: a decade of progress
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
Cooperative scans: dynamic bandwidth sharing in a DBMS
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
Breaking the memory wall in MonetDB
Communications of the ACM - Surviving the data deluge
The Database Architecture Jigsaw Puzzle
ICDE '08 Proceedings of the 2008 IEEE 24th International Conference on Data Engineering
An architecture for recycling intermediates in a column-store
Proceedings of the 2009 ACM SIGMOD International Conference on Management of data
Spinning relations: high-speed networks for distributed join processing
Proceedings of the Fifth International Workshop on Data Management on New Hardware
Minimizing the Hidden Cost of RDMA
ICDCS '09 Proceedings of the 2009 29th IEEE International Conference on Distributed Computing Systems
Database architecture evolution: mammals flourished long before dinosaurs became extinct
Proceedings of the VLDB Endowment
The Data Cyclotron query processing scheme
Proceedings of the 13th International Conference on Extending Database Technology
The DataPath system: a data-centric analytic processing engine for large data warehouses
Proceedings of the 2010 ACM SIGMOD International Conference on Management of data
An architecture for recycling intermediates in a column-store
ACM Transactions on Database Systems (TODS)
Hi-index | 0.00 |
A grand challenge of distributed query processing is to devise a self-organizing architecture which exploits all hardware resources optimally to manage the database hot set, minimize query response time, and maximize throughput without single point global coordination. The Data Cyclotron architecture [Goncalves and Kersten 2010] addresses this challenge using turbulent data movement through a storage ring built from distributed main memory and capitalizing on the functionality offered by modern remote-DMA network facilities. Queries assigned to individual nodes interact with the storage ring by picking up data fragments, which are continuously flowing around, that is, the hot set. The storage ring is steered by the Level Of Interest (LOI) attached to each data fragment, which represents the cumulative query interest as it passes around the ring multiple times. A fragment with LOI below a given threshold, inversely proportional to the ring load, is pulled out to free up resources. This threshold is dynamically adjusted in a fully distributed manner based on ring characteristics and locally observed query behavior. It optimizes resource utilization by keeping the average data access latency low. The approach is illustrated using an extensive and validated simulation study. The results underpin the fragment hot set management robustness in turbulent workload scenarios. A fully functional prototype of the proposed architecture has been implemented using modest extensions to MonetDB and runs within a multirack cluster equipped with Infiniband. Extensive experimentation using both microbenchmarks and high-volume workloads based on TPC-H demonstrates its feasibility. The Data Cyclotron architecture and experiments open a new vista for modern distributed database architectures with a plethora of new research challenges.