Applied multivariate statistical analysis
Applied multivariate statistical analysis
Measuring computer performance: a practitioner's guide
Measuring computer performance: a practitioner's guide
Adaptive optimization in the Jalapeño JVM
OOPSLA '00 Proceedings of the 15th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
An algorithm for parallel incremental compaction
Proceedings of the 3rd international symposium on Memory management
In or out?: putting write barriers in their place
Proceedings of the 3rd international symposium on Memory management
Online feedback-directed optimization of Java
OOPSLA '02 Proceedings of the 17th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
How java programs interact with virtual machines at the microarchitectural level
OOPSLA '03 Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications
Mostly concurrent garbage collection revisited
OOPSLA '03 Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications
Mark-copy: fast copying GC with less space overhead
OOPSLA '03 Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications
Ulterior reference counting: fast garbage collection without a long wait
OOPSLA '03 Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications
Oil and Water? High Performance Garbage Collection in Java with MMTk
Proceedings of the 26th International Conference on Software Engineering
Myths and realities: the performance impact of garbage collection
Proceedings of the joint international conference on Measurement and modeling of computer systems
The garbage collection advantage: improving program locality
OOPSLA '04 Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
Vertical profiling: understanding the behavior of object-priented applications
OOPSLA '04 Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
Reducing generational copy reserve overhead with fallback compaction
Proceedings of the 5th international symposium on Memory management
Proceedings of the 5th international symposium on Memory management
Improving locality with parallel hierarchical copying GC
Proceedings of the 5th international symposium on Memory management
Program-level adaptive memory management
Proceedings of the 5th international symposium on Memory management
Relative factors in performance analysis of Java virtual machines
Proceedings of the 2nd international conference on Virtual execution environments
Javana: a system for building customized Java program analysis tools
Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications
The DaCapo benchmarks: java benchmarking development and analysis
Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications
Using hardware performance monitors to understand the behavior of java applications
VM'04 Proceedings of the 3rd conference on Virtual Machine Research And Technology Symposium - Volume 3
Using hpm-sampling to drive dynamic compilation
Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems and applications
Adding rigorous statistics to the Java benchmarker's toolbox
Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion
Lightweight virtual machine support for AspectJ
Proceedings of the 7th international conference on Aspect-oriented software development
Wake up and smell the coffee: evaluation methodology for the 21st century
Communications of the ACM - Designing games with a purpose
Interprocedural query extraction for transparent persistence
Proceedings of the 23rd ACM SIGPLAN conference on Object-oriented programming systems languages and applications
Java performance evaluation through rigorous replay compilation
Proceedings of the 23rd ACM SIGPLAN conference on Object-oriented programming systems languages and applications
Producing wrong data without doing anything obviously wrong!
Proceedings of the 14th international conference on Architectural support for programming languages and operating systems
Influence of program inputs on the selection of garbage collectors
Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments
Empirical Performance Models for Java Workloads
ARCS '09 Proceedings of the 22nd International Conference on Architecture of Computing Systems
Chunking parallel loops in the presence of synchronization
Proceedings of the 23rd international conference on Supercomputing
Raced profiles: efficient selection of competing compiler optimizations
Proceedings of the 2009 ACM SIGPLAN/SIGBED conference on Languages, compilers, and tools for embedded systems
Keep It Small, Keep It Real: Efficient Run-Time Verification of Web Service Compositions
FMOODS '09/FORTE '09 Proceedings of the Joint 11th IFIP WG 6.1 International Conference FMOODS '09 and 29th IFIP WG 6.1 International Conference FORTE '09 on Formal Techniques for Distributed Systems
The study and handling of program inputs in the selection of garbage collectors
ACM SIGOPS Operating Systems Review
Allocation wall: a limiting factor of Java applications on emerging multi-core platforms
Proceedings of the 24th ACM SIGPLAN conference on Object oriented programming systems languages and applications
VM performance evaluation with functional models: an optimist's outlook
Proceedings of the Third Workshop on Virtual Machines and Intermediate Languages
Automated performance assessment for service-oriented middleware: a case study on BPEL engines
Proceedings of the 19th international conference on World wide web
Automated just-in-time compiler tuning
Proceedings of the 8th annual IEEE/ACM international symposium on Code generation and optimization
Evaluating the accuracy of Java profilers
PLDI '10 Proceedings of the 2010 ACM SIGPLAN conference on Programming language design and implementation
Local redundant polymorphism query elimination
Proceedings of the 8th International Conference on the Principles and Practice of Programming in Java
Reducing task creation and termination overhead in explicitly parallel programs
Proceedings of the 19th international conference on Parallel architectures and compilation techniques
Workshop on experimental evaluation of software and systems in computer science (Evaluate 2010)
Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion
JCoBox: generalizing active objects to concurrent components
ECOOP'10 Proceedings of the 24th European conference on Object-oriented programming
Late binding of AspectJ advice
TOOLS'10 Proceedings of the 48th international conference on Objects, models, components, patterns
Allocation removal by partial evaluation in a tracing JIT
Proceedings of the 20th ACM SIGPLAN workshop on Partial evaluation and program manipulation
THOR: a performance analysis tool for java applications running on multicore systems
IBM Journal of Research and Development
An empirical assessment of approaches to distributed enforcement in role-based access control (RBAC)
Proceedings of the first ACM conference on Data and application security and privacy
Looking back on the language and hardware revolutions: measured power, performance, and scaling
Proceedings of the sixteenth international conference on Architectural support for programming languages and operating systems
Polymorphic bytecode instrumentation
Proceedings of the tenth international conference on Aspect-oriented software development
A comprehensive evaluation of object scanning techniques
Proceedings of the international symposium on Memory management
Unique factorization domains in the java computer algebra system
ADG'08 Proceedings of the 7th international conference on Automated deduction in geometry
A generic parallel collection framework
Euro-Par'11 Proceedings of the 17th international conference on Parallel processing - Volume Part II
Automated GUI performance testing
Software Quality Control
Repeatability, reproducibility, and rigor in systems research
EMSOFT '11 Proceedings of the ninth ACM international conference on Embedded software
A step towards transparent integration of input-consciousness into dynamic program optimizations
Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications
A literate experimentation manifesto
Proceedings of the 10th SIGPLAN symposium on New ideas, new paradigms, and reflections on programming and software
Runtime feedback in a meta-tracing JIT for efficient dynamic languages
Proceedings of the 6th Workshop on Implementation, Compilation, Optimization of Object-Oriented Languages, Programs and Systems
Cyclic reference counting by typed reference fields
Computer Languages, Systems and Structures
Compiler mitigations for time attacks on modern x86 processors
ACM Transactions on Architecture and Code Optimization (TACO) - HIPEAC Papers
Instance-based XML data binding for mobile devices
Proceedings of the Third International Workshop on Middleware for Pervasive Mobile and Embedded Computing
Space-based multi-core programming in Java
Proceedings of the 9th International Conference on Principles and Practice of Programming in Java
A microbenchmark case study and lessons learned
Proceedings of the compilation of the co-located workshops on DSM'11, TMC'11, AGERE!'11, AOOPES'11, NEAT'11, & VMIL'11
Concurrent tries with efficient non-blocking snapshots
Proceedings of the 17th ACM SIGPLAN symposium on Principles and Practice of Parallel Programming
Measurement and dynamical analysis of computer performance data
IDA'10 Proceedings of the 9th international conference on Advances in Intelligent Data Analysis
SPECjbb2012: updated metrics for a business benchmark
ICPE '12 Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering
Web workload generation challenges - an empirical investigation
Software—Practice & Experience
Predicting performance via automated feature-interaction detection
Proceedings of the 34th International Conference on Software Engineering
MSEPT'12 Proceedings of the 2012 international conference on Multicore Software Engineering, Performance, and Tools
TOOLS'12 Proceedings of the 50th international conference on Objects, Models, Components, Patterns
Exploring multi-threaded Java application performance on multicore hardware
Proceedings of the ACM international conference on Object oriented programming systems languages and applications
Automatically enhancing locality for tree traversals with traversal splicing
Proceedings of the ACM international conference on Object oriented programming systems languages and applications
Integrating task parallelism with actors
Proceedings of the ACM international conference on Object oriented programming systems languages and applications
Reifying and optimizing collection queries for modularity
Proceedings of the 3rd annual conference on Systems, programming, and applications: software for humanity
Folding of tagged single assignment values for memory-efficient parallelism
Euro-Par'12 Proceedings of the 18th international conference on Parallel Processing
Parallel gesture recognition with soft real-time guarantees
Proceedings of the 2nd edition on Programming systems, languages and applications based on actors, agents, and decentralized control abstractions
Efficient support of dynamic inheritance for class- and prototype-based languages
Journal of Systems and Software
R3: repeatability, reproducibility and rigor
ACM SIGPLAN Notices - Supplemental issue
A Transformation Framework for Optimizing Task-Parallel Programs
ACM Transactions on Programming Languages and Systems (TOPLAS)
Why you should care about quantile regression
Proceedings of the eighteenth international conference on Architectural support for programming languages and operating systems
STABILIZER: statistically sound performance evaluation
Proceedings of the eighteenth international conference on Architectural support for programming languages and operating systems
Reify your collection queries for modularity and speed!
Proceedings of the 12th annual international conference on Aspect-oriented software development
Performance potential of optimization phase selection during dynamic JIT compilation
Proceedings of the 9th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments
A framework for application guidance in virtual memory systems
Proceedings of the 9th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments
Java in the High Performance Computing arena: Research, practice and experience
Science of Computer Programming
Rigorous benchmarking in reasonable time
Proceedings of the 2013 international symposium on memory management
Automated root cause isolation of performance regressions during software development
Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering
DataMill: rigorous performance evaluation made easy
Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering
Parallelism profiling and wall-time prediction for multi-threaded applications
Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering
A novel obfuscation: class hierarchy flattening
FPS'12 Proceedings of the 5th international conference on Foundations and Practice of Security
Are your incoming aliases really necessary? counting the cost of object ownership
Proceedings of the 2013 International Conference on Software Engineering
Extending the JastAdd extensible Java compiler to Java 7
Proceedings of the 2013 International Conference on Principles and Practices of Programming on the Java Platform: Virtual Machines, Languages, and Tools
Storage strategies for collections in dynamically typed languages
Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications
Miniboxing: improving the speed to code size tradeoff in parametric polymorphism translations
Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications
Interprocedural strength reduction of critical sections in explicitly-parallel programs
PACT '13 Proceedings of the 22nd international conference on Parallel architectures and compilation techniques
Higher-Order reactive programming with incremental lists
ECOOP'13 Proceedings of the 27th European conference on Object-Oriented Programming
A study of performance variations in the Mozilla Firefox web browser
ACSC '13 Proceedings of the Thirty-Sixth Australasian Computer Science Conference - Volume 135
A performance cost evaluation of aspect weaving
ACSC '13 Proceedings of the Thirty-Sixth Australasian Computer Science Conference - Volume 135
Exploring single and multilevel JIT compilation policy for modern machines 1
ACM Transactions on Architecture and Code Optimization (TACO)
Cloud PARTE: elastic complex event processing based on mobile actors
Proceedings of the 2013 workshop on Programming based on actors, agents, and decentralized control
A decoupled non-SSA global register allocation using bipartite liveness graphs
ACM Transactions on Architecture and Code Optimization (TACO)
Warm-Up Simulation Methodology for HW/SW Co-Designed Processors
Proceedings of Annual IEEE/ACM International Symposium on Code Generation and Optimization
QuEval: beyond high-dimensional indexing à la carte
Proceedings of the VLDB Endowment
A hybrid class- and prototype-based object model to support language-neutral structural intercession
Information and Software Technology
Hi-index | 0.00 |
Java performance is far from being trivial to benchmark because it is affected by various factors such as the Java application, its input, the virtual machine, the garbage collector, the heap size, etc. In addition, non-determinism at run-time causes the execution time of a Java program to differ from run to run. There are a number of sources of non-determinism such as Just-In-Time (JIT) compilation and optimization in the virtual machine (VM) driven by timer-based method sampling, thread scheduling, garbage collection, and various. There exist a wide variety of Java performance evaluation methodologies usedby researchers and benchmarkers. These methodologies differ from each other in a number of ways. Some report average performance over a number of runs of the same experiment; others report the best or second best performance observed; yet others report the worst. Some iterate the benchmark multiple times within a single VM invocation; others consider multiple VM invocations and iterate a single benchmark execution; yet others consider multiple VM invocations and iterate the benchmark multiple times. This paper shows that prevalent methodologies can be misleading, and can even lead to incorrect conclusions. The reason is that the data analysis is not statistically rigorous. In this paper, we present a survey of existing Java performance evaluation methodologies and discuss the importance of statistically rigorous data analysis for dealing with non-determinism. We advocate approaches to quantify startup as well as steady-state performance, and, in addition, we provide the JavaStats software to automatically obtain performance numbers in a rigorous manner. Although this paper focuses on Java performance evaluation, many of the issues addressed in this paper also apply to other programming languages and systems that build on a managed runtime system.