Evaluating program analysis and testing tools with the RUGRAT random benchmark application generator

  • Authors:

  • Ishtiaque Hussain;Christoph Csallner;Mark Grechanik;Chen Fu;Qing Xie;Sangmin Park;Kunal Taneja;B. M. Mainul Hossain

  • Affiliations:

  • University of Texas at Arlington, USA;University of Texas at Arlington, USA;Accenture Technology Labs, USA / University of Illinois at Chicago, USA;Accenture Technology Labs, USA;Accenture Technology Labs, USA;Georgia Tech, USA;Accenture Technology Labs, USA / North Carolina State University, USA;University of Illinois at Chicago, USA

  • Venue:
  • Proceedings of the 2012 Workshop on Dynamic Analysis
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Benchmarks are heavily used in different areas of computer science to evaluate algorithms and tools. In program analysis and testing, open-source and commercial programs are routinely used as bench- marks to evaluate different aspects of algorithms and tools. Unfor- tunately, many of these programs are written by programmers who introduce different biases, not to mention that it is very difficult to find programs that can serve as benchmarks with high reproducibil- ity of results. We propose a novel approach for generating random benchmarks for evaluating program analysis and testing tools. Our approach uses stochastic parse trees, where language grammar production rules are assigned probabilities that specify the frequencies with which instantiations of these rules will appear in the generated pro- grams. We implemented our tool for Java and applied it to generate benchmarks with which we evaluated different program analysis and testing tools. Our tool was also implemented by a major soft- ware company for C++ and used by a team of developers to gener- ate benchmarks that enabled them to reproduce a bug in less than four hours.