Practical testing of a C99 compiler using output comparison

  • Authors:
  • Flash Sheridan

  • Affiliations:
  • Access Systems Americas, Inc. (formerly PalmSource, Inc.), 1188 East Arques Avenue, Sunnyvale, CA 94305, U.S.A.

  • Venue:
  • Software—Practice & Experience
  • Year:
  • 2007

Quantified Score

Hi-index 0.01

Visualization

Abstract

A simple technique is presented for testing a C99 compiler, bycomparing its output with the output from pre-existing tools. Theadvantage to this approach is that new test cases can be added inbulk from existing sources, reducing the need for in-depthinvestigation of correctness issues and for creating new test codeby hand. This technique was used in testing the PalmSource PalmOS® Cobalt ARM C/C++ cross-compiler for Palm-Powered®personal digital assistants, primarily for standards compliance andthe correct execution of generated code. The technique describedhere found several hundred bugs, mostly in our in-house code, butalso in longstanding high-quality front- and back-end code fromEdison Design Group and Apogee Software. It also found 18 bugs inthe GNU C compiler, as well as a bug specific to the Apple versionof GCC, a bug specific to the Suse version of GCC, and a dozen bugsin versions of GCC for the ARM processor, several of which werecritical. Copyright © 2007 John Wiley & Sons, Ltd.