Applied multivariate statistical analysis
Applied multivariate statistical analysis
A portable sampling-based profiler for Java virtual machines
Proceedings of the ACM 2000 conference on Java Grande
Measuring computer performance: a practitioner's guide
Measuring computer performance: a practitioner's guide
Adaptive optimization in the Jalapeño JVM
OOPSLA '00 Proceedings of the 15th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
Using annotations to reduce dynamic optimization time
Proceedings of the ACM SIGPLAN 2001 conference on Programming language design and implementation
How java programs interact with virtual machines at the microarchitectural level
OOPSLA '03 Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications
Mark-copy: fast copying GC with less space overhead
OOPSLA '03 Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications
Inducing heuristics to decide whether to schedule
Proceedings of the ACM SIGPLAN 2004 conference on Programming language design and implementation
Oil and Water? High Performance Garbage Collection in Java with MMTk
Proceedings of the 26th International Conference on Software Engineering
Myths and realities: the performance impact of garbage collection
Proceedings of the joint international conference on Measurement and modeling of computer systems
The garbage collection advantage: improving program locality
OOPSLA '04 Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
MC2: high-performance garbage collection for memory-constrained environments
OOPSLA '04 Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
Vertical profiling: understanding the behavior of object-priented applications
OOPSLA '04 Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
Dynamic selection of application-specific garbage collectors
Proceedings of the 4th international symposium on Memory management
Automatic heap sizing: taking real memory into account
Proceedings of the 4th international symposium on Memory management
Proceedings of the 4th international symposium on Memory management
Design and evaluation of dynamic optimizations for a Java just-in-time compiler
ACM Transactions on Programming Languages and Systems (TOPLAS)
Continuous Path and Edge Profiling
Proceedings of the 38th annual IEEE/ACM International Symposium on Microarchitecture
Experiences with Multi-threading and Dynamic Class Loading in a Java Just-In-Time Compiler
Proceedings of the International Symposium on Code Generation and Optimization
Free-Me: a static analysis for automatic individual object reclamation
Proceedings of the 2006 ACM SIGPLAN conference on Programming language design and implementation
Relative factors in performance analysis of Java virtual machines
Proceedings of the 2nd international conference on Virtual execution environments
Javana: a system for building customized Java program analysis tools
Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications
The DaCapo benchmarks: java benchmarking development and analysis
Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications
Replay compilation: improving debuggability of a just-in-time compiler
Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications
Bell: bit-encoding online memory leak detection
Proceedings of the 12th international conference on Architectural support for programming languages and operating systems
ACM Transactions on Programming Languages and Systems (TOPLAS)
Online optimizations driven by hardware performance monitoring
Proceedings of the 2007 ACM SIGPLAN conference on Programming language design and implementation
Using hardware performance monitors to understand the behavior of java applications
VM'04 Proceedings of the 3rd conference on Virtual Machine Research And Technology Symposium - Volume 3
The java hotspotTM server compiler
JVM'01 Proceedings of the 2001 Symposium on JavaTM Virtual Machine Research and Technology Symposium - Volume 1
Statistically rigorous java performance evaluation
Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems and applications
Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems and applications
Using hpm-sampling to drive dynamic compilation
Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems and applications
Correcting the dynamic call graph using control-flow constraints
CC'07 Proceedings of the 16th international conference on Compiler construction
Proceedings of the 14th international conference on Architectural support for programming languages and operating systems
Reducing performance non-determinism via cache-aware page allocation strategies
Proceedings of the first joint WOSP/SIPEW international conference on Performance engineering
Efficient, context-sensitive detection of real-world semantic attacks
PLAS '10 Proceedings of the 5th ACM SIGPLAN Workshop on Programming Languages and Analysis for Security
Repeatability, reproducibility, and rigor in systems research
EMSOFT '11 Proceedings of the ninth ACM international conference on Embedded software
Proceedings of the compilation of the co-located workshops on DSM'11, TMC'11, AGERE!'11, AOOPES'11, NEAT'11, & VMIL'11
Split register allocation: linear complexity without the performance penalty
HiPEAC'10 Proceedings of the 5th international conference on High Performance Embedded Architectures and Compilers
RATA: rapid atomic type analysis by abstract interpretation – application to javascript optimization
CC'10/ETAPS'10 Proceedings of the 19th joint European conference on Theory and Practice of Software, international conference on Compiler Construction
R3: repeatability, reproducibility and rigor
ACM SIGPLAN Notices - Supplemental issue
Rigorous benchmarking in reasonable time
Proceedings of the 2013 international symposium on memory management
Proceedings of the ACM International Conference on Computing Frontiers
Warm-Up Simulation Methodology for HW/SW Co-Designed Processors
Proceedings of Annual IEEE/ACM International Symposium on Code Generation and Optimization
Hi-index | 0.00 |
A managed runtime environment, such as the Java virtual machine, is non-trivial to benchmark. Java performance is affected in various complex ways by the application and its input, as well as by the virtual machine (JIT optimizer, garbage collector, thread scheduler, etc.). In addition, non-determinism due to timer-based sampling for JIT optimization, thread scheduling, and various system effects further complicate the Java performance benchmarking process. Replay compilation is a recently introduced Java performance analysis methodology that aims at controlling non-determinism to improve experimental repeatability. The key idea of replay compilation is to control the compilation load during experimentation by inducing a pre-recorded compilation plan at replay time. Replay compilation also enables teasing apart performance effects of the application versus the virtual machine. This paper argues that in contrast to current practice which uses a single compilation plan at replay time, multiple compilation plans add statistical rigor to the replay compilation methodology. By doing so, replay compilation better accounts for the variability observed in compilation load across compilation plans. In addition, we propose matched-pair comparison for statistical data analysis. Matched-pair comparison considers the performance measurements per compilation plan before and after an innovation of interest as a pair, which enables limiting the number of compilation plans needed for accurate performance analysis compared to statistical analysis assuming unpaired measurements.