Automated Detection of Performance Regressions: The Mono Experience

Authors:
Tomas Kalibera;Lubomir Bulej;Petr Tuma
Affiliations:
Distributed Systems Research Group, Department of Software Engineering Faculty of Mathematics and Physics, Charles University, Malostranske nam. Prague, Czech Republic;Distributed Systems Research Group, Department of Software Engineering Faculty of Mathematics and Physics, Charles University, Malostranske nam. Prague, Czech Republic;Institute of Computer Science, Czech Academy of Sciences Pod Vodarenskou vezi 2, 182 07 Prague, Czech Republic
Venue:
MASCOTS '05 Proceedings of the 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems
Year:
2005

Citing 0
Cited 5

Reducing performance non-determinism via cache-aware page allocation strategies

Proceedings of the first joint WOSP/SIPEW international conference on Performance engineering
Repeatability, reproducibility, and rigor in systems research

EMSOFT '11 Proceedings of the ninth ACM international conference on Embedded software
Precise regression benchmarking with random effects: improving mono benchmark results

EPEW'06 Proceedings of the Third European conference on Formal Methods and Stochastic Models for Performance Evaluation
R3: repeatability, reproducibility and rigor

ACM SIGPLAN Notices - Supplemental issue
Where has all my memory gone?: determining memory characteristics of product variants using virtual-machine-level monitoring

Proceedings of the Eighth International Workshop on Variability Modelling of Software-Intensive Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

Engineering a large software project involves tracking the impact of development and maintenance changes on the software performance. An approach for tracking the impact is regression benchmarking, which involves automated benchmarking and evaluation of performance at regular intervals. Regression benchmarking must tackle the nondeterminism inherent to contemporary computer systems and execution environments and the impact of the nondeterminism on the results. On the example of a fully automated regression benchmarking environment for the Mono opensource project, we show how the problems associated with nondeterminism can be tackled using statistical methods.