A microbenchmark case study and lessons learned

Authors:
Joseph Yossi Gil;Keren Lenz;Yuval Shimron
Affiliations:
Israel Institute of Technology, Technion City, Haifa, Israel;Israel Institute of Technology, Technion City, Haifa, Israel;Israel Institute of Technology, Technion City, Haifa, Israel
Venue:
Proceedings of the compilation of the co-located workshops on DSM'11, TMC'11, AGERE!'11, AOOPES'11, NEAT'11, & VMIL'11
Year:
2011

Citing 10
Cited 1

The Java programming language (2nd ed.)

The Java programming language (2nd ed.)
Online feedback-directed optimization of Java

OOPSLA '02 Proceedings of the 17th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
How java programs interact with virtual machines at the microarchitectural level

OOPSLA '03 Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications
The garbage collection advantage: improving program locality

OOPSLA '04 Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
Method-level phase behavior in java workloads

OOPSLA '04 Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
The DaCapo benchmarks: java benchmarking development and analysis

Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications
Statistically rigorous java performance evaluation

Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems and applications
Wake up and smell the coffee: evaluation methodology for the 21st century

Communications of the ACM - Designing games with a purpose
Languages and performance engineering: method, instrumentation, and pedagogy

ACM SIGPLAN Notices
Producing wrong data without doing anything obviously wrong!

Proceedings of the 14th international conference on Architectural support for programming languages and operating systems

Smaller footprint for java collections

ECOOP'12 Proceedings of the 26th European conference on Object-Oriented Programming

Quantified Score

Hi-index	0.00

Visualization

Abstract

The extra abstraction layer posed by the virtual machine, the JIT compilation cycles and the asynchronous garbage collection are the main reasons that make the benchmarking of Java code a delicate task. The primary weapon in battling these is replication: "billions and billions of runs", is phrase sometimes used by practitioners. This paper describes a case study, which consumed hundreds of hours of CPU time, and tries to characterize the inconsistencies in the results we encountered.