Note on generalization in experimental algorithmics

  • Authors:
  • Naren Ramakrishnan;Raúl E. Valdés-Pérez

  • Affiliations:
  • Virginia Polytechnic Institute and State Univ., Blacksburg;Carnegie Mellon Univ., Pittsburgh, PA

  • Venue:
  • ACM Transactions on Mathematical Software (TOMS)
  • Year:
  • 2000

Quantified Score

Hi-index 0.00

Visualization

Abstract

A recurring theme in mathematical software evaluation is the generalization of rankings of algorithms on test problems to build knowledge-based recommender systems for algorithm selection. A key issue is to profile algorithms in terms of the qualitative characteristics of benchmark problems. In this methodological note, we adapt a novel all-pairs algorithm for the profiling task; given performance rankings for m algorithms on n problem instances, each described with p features, identify a (minimal) subset of p that is useful for assessing the selective superiority of an algorithm over another, for all pairs of m algorithms. We show how techniques presented in the mathematical software literature are inadequate for such profiling purposes. In conclusion, we also address various statistical issues underlying the effective application of this technique.