Achieving COSMOS: a metric for determining when to give up and when to reach for the stars

Authors:
Emma Tosch;Lee Spector
Affiliations:
University of Massachusetts, Amherst, MA, USA;Hampshire College, Amherst, MA, USA
Venue:
Proceedings of the 14th annual conference companion on Genetic and evolutionary computation
Year:
2012

Citing 10
Cited 1

What Makes a Problem GP-Hard? Analysis of a Tunably Difficult Problem in Genetic Programming

Genetic Programming and Evolvable Machines
Is The Perfect The Enemy Of The Good?

GECCO '02 Proceedings of the Genetic and Evolutionary Computation Conference
Pattern Classification (2nd Edition)

Pattern Classification (2nd Edition)
The reliability of confidence intervals for computational effort comparisons

Proceedings of the 9th annual conference on Genetic and evolutionary computation
Is "best-so-far" a good algorithmic performance metric?

Proceedings of the 10th annual conference on Genetic and evolutionary computation
On Improving Generalisation in Genetic Programming

EuroGP '09 Proceedings of the 12th European Conference on Genetic Programming
An investigation into the sensitivity of genetic programming to the frequency of leaf selection during subtree crossover

GECCO '96 Proceedings of the 1st annual conference on Genetic and evolutionary computation
More on computational effort statistics for genetic programming

EuroGP'03 Proceedings of the 6th European conference on Genetic programming
Confidence intervals for computational effort comparisons

EuroGP'07 Proceedings of the 10th European conference on Genetic programming
Statistical distribution of generation-to-success in GP: application to model accumulated success probability

EuroGP'11 Proceedings of the 14th European conference on Genetic programming

Autoconstructive evolution for structural problems

Proceedings of the 14th annual conference companion on Genetic and evolutionary computation

Quantified Score

Hi-index	0.00

Visualization

Abstract

The utility of current metrics used in genetic programming (GP) systems, such as computational effort and mean-best-fitness, varies depending upon the problem and the resource that needs to be optimized. Inferences about the underlying system can only be made when a sufficient number of runs are performed to estimate the relevant metric within some confidence interval. This paper proposes a new algorithm for determining the minimum number of independent runs needed to make inferences about a GP system. As such, we view our algorithm as a meta-metric that should be satisfied before any inferences about a system are made. We call this metric COSMOS, as it estimates the number of independent runs needed to achieve the Convergence Of Sample Means Of the Order Statistics. It is agnostic to the underlying GP system and can be used to evaluate extant performance metrics, as well as problem difficulty. We suggest ways for which COSMOS may be used to identify problems for which GP may be uniquely qualified to solve.