Genetic programming needs better benchmarks

  • Authors:

  • James McDermott;David R. White;Sean Luke;Luca Manzoni;Mauro Castelli;Leonardo Vanneschi;Wojciech Jaskowski;Krzysztof Krawiec;Robin Harper;Kenneth De Jong;Una-May O'Reilly

  • Affiliations:

  • Massachusetts Institute of Technology, Cambridge, MA, USA;University of Glasgow, Glasgow, United Kingdom;George Mason University, Fairfax, VA, USA;Universita degli Studi di Milano - Bicocca, Milan, Italy;Universita degli Studi di Milano - Bicocca, Milan, Italy;Universidade Nova de Lisboa, Lisbon, Portugal;Poznan University of Technology, Poznan, Poland;Poznan University of Technology, Poznan, Poland;University of Sydney, Sydney, Australia;George Mason University, Fairfax, VA, USA;Massachusetts Institute of Technology, Cambridge, MA, USA

  • Venue:
  • Proceedings of the 14th annual conference on Genetic and evolutionary computation
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Genetic programming (GP) is not a field noted for the rigor of its benchmarking. Some of its benchmark problems are popular purely through historical contingency, and they can be criticized as too easy or as providing misleading information concerning real-world performance, but they persist largely because of inertia and the lack of good alternatives. Even where the problems themselves are impeccable, comparisons between studies are made more difficult by the lack of standardization. We argue that the definition of standard benchmarks is an essential step in the maturation of the field. We make several contributions towards this goal. We motivate the development of a benchmark suite and define its goals; we survey existing practice; we enumerate many candidate benchmarks; we report progress on reference implementations; and we set out a concrete plan for gathering feedback from the GP community that would, if adopted, lead to a standard set of benchmarks.