Towards robust distributed systems (abstract)
Proceedings of the nineteenth annual ACM symposium on Principles of distributed computing
Data access patterns in the Amazon.com technology platform
VLDB '07 Proceedings of the 33rd international conference on Very large data bases
Communications of the ACM - Web science
Data management projects at Google
ACM SIGMOD Record
Proceedings of the 2008 ACM SIGMOD international conference on Management of data
An evaluation of alternative architectures for transaction processing in the cloud
Proceedings of the 2010 ACM SIGMOD International Conference on Management of data
Runtime measurements in the cloud: observing, analyzing, and reducing variance
Proceedings of the VLDB Endowment
A data generator for cloud-scale benchmarking
TPCTC'10 Proceedings of the Second TPC technology conference on Performance evaluation, measurement and characterization of complex systems
Towards using and improving the NAS parallel benchmarks: a parallel patterns approach
Proceedings of the 2010 Workshop on Parallel Programming Patterns
Aspects of data-intensive cloud computing
From active data management to event-based systems and more
Autonomic benchmarking for cloud infrastructures: an economic optimization model
Proceedings of the 1st ACM/IEEE workshop on Autonomic computing in economics
How a consumer can measure elasticity for cloud platforms
ICPE '12 Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering
Performance Evaluation of Range Queries in Key Value Stores
Journal of Grid Computing
When free is not really free: what does it cost to run a database workload in the cloud?
TPCTC'11 Proceedings of the Third TPC Technology conference on Topics in Performance Evaluation, Measurement and Characterization
Benchmarking cloud security level agreements using quantitative policy trees
Proceedings of the 2012 ACM Workshop on Cloud computing security workshop
On a Catalogue of Metrics for Evaluating Commercial Cloud Services
GRID '12 Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing
Hardware-in-the-loop simulation for automated benchmarking of cloud infrastructures
Proceedings of the Winter Simulation Conference
A framework for ranking of cloud computing services
Future Generation Computer Systems
Position paper: cloud system deployment and performance evaluation tools for distributed databases
Proceedings of the 2013 international workshop on Hot topics in cloud services
Survey Cloud monitoring: A survey
Computer Networks: The International Journal of Computer and Telecommunications Networking
Multi-tenancy performance benchmark for web application platforms
ICWE'13 Proceedings of the 13th international conference on Web Engineering
UpSizeR: Synthetically scaling an empirical relational database
Information Systems
Context-aware and automatic configuration of mobile devices in cloud-enabled ubiquitous computing
Personal and Ubiquitous Computing
| Hi-index | 0.00 |
Traditionally, the goal of benchmarking a software system is to evaluate its performance under a particular workload for a fixed configuration. The most prominent examples for evaluating transactional database systems as well as other components on top (such as a application-servers or web-servers) are the various TPC benchmarks. In this paper we argue that traditional benchmarks (like the TPC benchmarks) are not sufficient for analyzing the novel cloud services. Moreover, we present some initial ideas how such a new benchmark should look like that fits better to the characteristics of cloud computing (e.g., scalability, pay-per-use and fault-tolerance). The main challenge of such a new benchmark is to make the reported results comparable because different providers offer different services with different capabilities and guarantees.