How not to lie with statistics: the correct way to summarize benchmark results

Authors:
Philip J. Fleming;John J. Wallace
Affiliations:
AT&T Information Systems, Naperville, IL;The Foxboro Company, Foxboro, MA
Venue:
Communications of the ACM - The MIT Press scientific computation series
Year:
1986

Citing 1
Cited 39

Re-evaluation of the RISC I

ACM SIGARCH Computer Architecture News

Performance evaluation of on-chip register and cache organizations

ISCA '88 Proceedings of the 15th Annual International Symposium on Computer architecture
Characterizing computer performance with a single number

Communications of the ACM
ParcBench: a benchmark for shared-memory architectures

ACM SIGARCH Computer Architecture News
Performance analysis of high-speed computers

Proceedings of the 1989 ACM/IEEE conference on Supercomputing
Cache performance of the integer SPEC benchmarks on a RISC

ACM SIGARCH Computer Architecture News
Performance variation across benchmark suites

ACM SIGMETRICS Performance Evaluation Review
Performance variation across benchmark suites

ACM SIGARCH Computer Architecture News
Performance evaluation of a decoded instruction cache for variable instruction-length computers

ISCA '92 Proceedings of the 19th annual international symposium on Computer architecture
Using architectural “families” to increase FPGA speed and density

FPGA '95 Proceedings of the 1995 ACM third international symposium on Field-programmable gate arrays
The memory fragmentation problem: solved?

Proceedings of the 1st international symposium on Memory management
Simple vector microprocessors for multimedia applications

MICRO 31 Proceedings of the 31st annual ACM/IEEE international symposium on Microarchitecture
Performance characteristics of microcomputer Prolog implementations

Proceedings of the 1986 ACM SIGSMALL/PC symposium on Small systems
SPEC as a Performance Evaluation Measure

Computer
Accurate Unix Benchmarking: Art, Science, or Black Magic?

IEEE Micro
Performance Evaluation of a Decoded Instruction Cache for Variable Instruction Length Computers

IEEE Transactions on Computers
Software Measurement: A Necessary Scientific Basis

IEEE Transactions on Software Engineering
Design and Results of the Tableaux-99 Non-classical (Modal) Systems Comparison

TABLEAUX '99 Proceedings of the International Conference on Automated Reasoning with Analytic Tableaux and Related Methods
How to fake an RSA signature by encoding modular root finding as a SAT problem

Discrete Applied Mathematics - The renesse issue on satisfiability
Secrets of the Glasgow Haskell Compiler inliner

Journal of Functional Programming
War of the benchmark means: time for a truce

ACM SIGARCH Computer Architecture News
Requirements-driven database systems benchmark method

Decision Support Systems
The harmonic or geometric mean: does it really matter?

ACM SIGARCH Computer Architecture News
An optimal multiprocessor combinatorial auction solver

Computers and Operations Research
A Streamlined and Generalized Analysis of Chromatin ImmunoPrecipitation Paired-End diTag Data

ICCS '08 Proceedings of the 8th international conference on Computational Science, Part III
A Twofold Distributed Game-Tree Search Approach Using Interconnected Clusters

Euro-Par '08 Proceedings of the 14th international Euro-Par conference on Parallel Processing
From State-of-the-Art Static Fleet Assignment to Flexible Stochastic Planning of the Future

Algorithmics of Large and Complex Networks
Probabilistic performance profiles for the experimental evaluation of stochastic algorithms

Proceedings of the 12th annual conference on Genetic and evolutionary computation
Directed model checking for B: an evaluation and new techniques

SBMF'10 Proceedings of the 13th Brazilian conference on Formal methods: foundations and applications
Language-independent sandboxing of just-in-time compilation and self-modifying code

Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation
How to measure useful, sustained performance

State of the Practice Reports
Parallelism for perturbation management and robust plans

Euro-Par'05 Proceedings of the 11th international Euro-Par conference on Parallel Processing
An optimal filtering algorithm for table constraints

CP'12 Proceedings of the 18th international conference on Principles and Practice of Constraint Programming
Federating queries in SPARQL 1.1: Syntax, semantics and evaluation

Web Semantics: Science, Services and Agents on the World Wide Web
Assessing computer performance with stocs

Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering
Quantitative evaluation of soft error injection techniques for robust system design

Proceedings of the 50th Annual Design Automation Conference
Storage strategies for collections in dynamically typed languages

Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications
Exploiting phase inter-dependencies for faster iterative compiler optimization phase order searches

Proceedings of the 2013 International Conference on Compilers, Architectures and Synthesis for Embedded Systems
Optimal and efficient filtering algorithms for table constraints

Constraints
Static analysis of multi-core TDMA resource arbitration delays

Real-Time Systems

Quantified Score

Hi-index	0.02

Visualization

Abstract

Using the arithmetic mean to summarize normalized benchmark results leads to mistaken conclusions that can be avoided by using the preferred method: the geometric mean.