Managing operational business intelligence workloads
ACM SIGOPS Operating Systems Review
Query interactions in database workloads
Proceedings of the Second International Workshop on Testing Database Systems
ParaTimer: a progress indicator for MapReduce DAGs
Proceedings of the 2010 ACM SIGMOD International Conference on Management of data
PeerWatch: a fault detection and diagnosis tool for virtualized consolidation systems
Proceedings of the 7th international conference on Autonomic computing
Reflective control for an elastic cloud application: an automated experiment workbench
HotCloud'09 Proceedings of the 2009 conference on Hot topics in cloud computing
HotOS'09 Proceedings of the 12th conference on Hot topics in operating systems
A case for machine learning to optimize multicore performance
HotPar'09 Proceedings of the First USENIX conference on Hot topics in parallelism
SLA-tree: a framework for efficiently supporting SLA-based decisions in cloud computing
Proceedings of the 14th International Conference on Extending Database Technology
Predicting completion times of batch query workloads using interaction-aware models and simulation
Proceedings of the 14th International Conference on Extending Database Technology
Predicting system performance for multi-tenant database workloads
Proceedings of the Fourth International Workshop on Testing Database Systems
Performance prediction for concurrent database workloads
Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
Vrisha: using scaling properties of parallel programs for bug detection and localization
Proceedings of the 20th international symposium on High performance distributed computing
A bayesian approach to online performance modeling for database appliances using gaussian models
Proceedings of the 8th ACM international conference on Autonomic computing
iCBS: incremental cost-based scheduling under piecewise linear SLAs
Proceedings of the VLDB Endowment
Interaction-aware scheduling of report-generation workloads
The VLDB Journal — The International Journal on Very Large Data Bases
CloudScale: elastic resource scaling for multi-tenant cloud systems
Proceedings of the 2nd ACM Symposium on Cloud Computing
ActiveSLA: a profit-oriented admission control framework for database-as-a-service providers
Proceedings of the 2nd ACM Symposium on Cloud Computing
Design implications for enterprise storage systems via multi-dimensional trace analysis
SOSP '11 Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles
Aggregation strategies for columnar in-memory databases in a mixed workload
Proceedings of the 4th workshop on Workshop for Ph.D. students in information & knowledge management
Advances and challenges in log analysis
Communications of the ACM
PIQL: success-tolerant query processing in the cloud
Proceedings of the VLDB Endowment
Advances and Challenges in Log Analysis
Queue - Log Analysis
Performance evaluation of scheduling algorithms for database services with soft and hard SLAs
Proceedings of the second international workshop on Data intensive computing in the clouds
A statistical approach towards robust progress estimation
Proceedings of the VLDB Endowment
Managing dynamic mixed workloads for operational business intelligence
DNIS'10 Proceedings of the 6th international conference on Databases in Networked Information Systems
PerfXplain: debugging MapReduce job performance
Proceedings of the VLDB Endowment
Using computer simulation to predict the performance of multithreaded programs
ICPE '12 Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering
Dynamic management of resources and workloads for RDBMS in cloud: a control-theoretic approach
PhD '12 Proceedings of the on SIGMOD/PODS 2012 PhD Symposium
Computing resource prediction for mapreduce applications using decision tree
APWeb'12 Proceedings of the 14th Asia-Pacific international conference on Web Technologies and Applications
Predicting performance via automated feature-interaction detection
Proceedings of the 34th International Conference on Software Engineering
Robust estimation of resource consumption for SQL queries using statistical techniques
Proceedings of the VLDB Endowment
DBMS metrology: measuring query time
Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
Performance and resource modeling in highly-concurrent OLTP workloads
Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
Workload management for big data analytics
Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
Towards building performance models for data-intensive workloads in public clouds
Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering
Mantis: automatic performance prediction for smartphone applications
USENIX ATC'13 Proceedings of the 2013 USENIX conference on Annual Technical Conference
Towards predicting query execution time for concurrent and dynamic database workloads
Proceedings of the VLDB Endowment
Distribution-based query scheduling
Proceedings of the VLDB Endowment
PREDIcT: towards predicting the runtime of large scale iterative analytics
Proceedings of the VLDB Endowment
Hi-index | 0.02 |
One of the most challenging aspects of managing a very large data warehouse is identifying how queries will behave before they start executing. Yet knowing their performance characteristics --- their runtimes and resource usage --- can solve two important problems. First, every database vendor struggles with managing unexpectedly long-running queries. When these long-running queries can be identified before they start, they can be rejected or scheduled when they will not cause extreme resource contention for the other queries in the system. Second, deciding whether a system can complete a given workload in a given time period (or a bigger system is necessary) depends on knowing the resource requirements of the queries in that workload. We have developed a system that uses machine learning to accurately predict the performance metrics of database queries whose execution times range from milliseconds to hours. For training and testing our system, we used both real customer queries and queries generated from an extended set of TPC-DS templates. The extensions mimic queries that caused customer problems. We used these queries to compare how accurately different techniques predict metrics such as elapsed time, records used, disk I/Os, and message bytes. The most promising technique was not only the most accurate, but also predicted these metrics simultaneously and using only information available prior to query execution. We validated the accuracy of this machine learning technique on a number of HP Neoview configurations. We were able to predict individual query elapsed time within 20% of its actual time for 85% of the test queries. Most importantly, we were able to correctly identify both the short and long-running (up to two hour) queries to inform workload management and capacity planning.