Why SpecInt95 should not be used to benchmark embedded systems tools

  • Authors:
  • Jakob Engblom

  • Affiliations:
  • Department of Computer Systems, Uppsala University, P.O. Box 325, SE-751 05 Uppsala, Sweden

  • Venue:
  • Proceedings of the ACM SIGPLAN 1999 workshop on Languages, compilers, and tools for embedded systems
  • Year:
  • 1999

Quantified Score

Hi-index 0.00

Visualization

Abstract

The SpecInt95 benchmark suite is often used to evaluate the performance of programming tools, including those used for embedded systems programming. Embedded applications, however, are often targeting 8- or 16-bit processors with limited functionality, whereas SpecInt95 has no particular target architecture and a bias towards 32-bit systems. Hence, there are reasons to question the use of SpecInt95 for the evaluation of tools for embedded systems.We present a comparative study of the static properties of a set of embedded application and the SpecInt95 benchmarks. The properties studied include: variable types, function argument lists, type of operations, and the use of local and global memory.The study provides-clear evidence that embedded applications and the SpecInt95 program suite differs significantly in several important areas. Hence, we conclude that using SpecInt95 to evaluate or compare tools for embedded systems is likely to be irrelevant or misleading, and that there is a clear need for a benchmark suite tailored for the embedded applications area.