How not to lie with statistics: the correct way to summarize benchmark results
Communications of the ACM - The MIT Press scientific computation series
Characterizing computer performance with a single number
Communications of the ACM
Computer architecture: a quantitative approach
Computer architecture: a quantitative approach
Computer benchmarks
ACM SIGARCH Computer Architecture News
Adapting the SPEC 2000 benchmark suite for simulation-based computer architecture research
Workload characterization of emerging computer applications
BTU: A Host Communication Benchmark
Computer
IEEE Micro
Performance Analysys of a CC-NUMAOperating System
IPDPS '01 Proceedings of the 15th International Parallel & Distributed Processing Symposium
A Study of the Allocation Behavior of the SPECjvm98 Java Benchmark
ECOOP '99 Proceedings of the 13th European Conference on Object-Oriented Programming
Empirical Study of Object-Layout Strategies and Optimization Techniques
ECOOP '00 Proceedings of the 14th European Conference on Object-Oriented Programming
Metrics for Parallel Job Scheduling and Their Convergence
JSSPP '01 Revised Papers from the 7th International Workshop on Job Scheduling Strategies for Parallel Processing
A Statistically Rigorous Approach for Improving Simulation Methodology
HPCA '03 Proceedings of the 9th International Symposium on High-Performance Computer Architecture
MisSPECulation: partial and misleading use of SPEC CPU2000 in computer architecture conferences
Proceedings of the 30th annual international symposium on Computer architecture
War of the benchmark means: time for a truce
ACM SIGARCH Computer Architecture News
Measuring Benchmark Similarity Using Inherent Program Characteristics
IEEE Transactions on Computers
Subsetting the SPEC CPU2006 benchmark suite
ACM SIGARCH Computer Architecture News
Analysis of redundancy and application balance in the SPEC CPU2006 benchmark suite
Proceedings of the 34th annual international symposium on Computer architecture
Scenario-oriented design for single-chip heterogeneous multiprocessors
IEEE Transactions on Very Large Scale Integration (VLSI) Systems
Characteristics of workloads using the pipeline programming model
ISCA'10 Proceedings of the 2010 international conference on Computer Architecture
Impact of pay-as-you-go cloud platforms on software pricing and development: a review and case study
ICCSA'12 Proceedings of the 12th international conference on Computational Science and Its Applications - Volume Part IV
| Hi-index | 4.10 |
System Performance Evaluation Cooperative (SPEC) measures are commonly considered the most effective indicators for evaluating engineering workstation performance. Indeed, these measures have gradually become a de facto industry standard for CPU performance evaluation. But do they truly provide added information value? The authors examine SPEC measures with the aim of learning their limitations and the areas in which caution should be exercised. They present results and evaluations of a comprehensive statistical analysis of SPEC rankings published in the SPEC Newsletter over a number of years. The major findings show that several programs in the benchmark suite contribute little additional information to the measures. In fact, the potency profile of the system can be demonstrated by just a few representative programs. Moreover, the method of aggregating the results to create a single measure (or two measures) leads to distortion in presenting the overall system performance. In evaluating the system, SPEC focuses on the CPU clock rate and thus may not reflect the performance of the entire system. This suggests that the added value of the SPEC information is not very high. The authors thus advise that SPEC should be employed cautiously with full awareness of its shortcomings.