A Critical Look at Experimental Evaluations of EBL
Machine Learning
Statistical Methods for Analyzing Speedup Learning Experiments
Machine Learning
Morphing: combining structure and randomness
AAAI '99/IAAI '99 Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence
Towards a universal test suite for combinatorial auction algorithms
Proceedings of the 2nd ACM conference on Electronic commerce
Artificial Intelligence - special issue on computational tradeoffs under bounded resources
A Racing Algorithm for Configuring Metaheuristics
GECCO '02 Proceedings of the Genetic and Evolutionary Computation Conference
A Bayesian Approach to Tackling Hard Computational Problems
UAI '01 Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence
Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions
CP '02 Proceedings of the 8th International Conference on Principles and Practice of Constraint Programming
Experimental Research in Evolutionary Computation: The New Experimentalism (Natural Computing Series)
Finding Optimal Algorithmic Parameters Using Derivative-Free Optimization
SIAM Journal on Optimization
Fine-Tuning of Algorithms Using Fractional Experimental Designs and Local Search
Operations Research
Boosting Verification by Automatic Tuning of Decision Procedures
FMCAD '07 Proceedings of the Formal Methods in Computer Aided Design
Empirical hardness models: Methodology and a case study on combinatorial auctions
Journal of the ACM (JACM)
An experimental investigation of model-based parameter optimisation: SPO and beyond
Proceedings of the 11th Annual conference on Genetic and evolutionary computation
Automatic algorithm configuration based on local search
AAAI'07 Proceedings of the 22nd national conference on Artificial intelligence - Volume 2
SATzilla: portfolio-based algorithm selection for SAT
Journal of Artificial Intelligence Research
SATenstein: automatically building local search SAT solvers from components
IJCAI'09 Proceedings of the 21st international jont conference on Artifical intelligence
ParamILS: an automatic algorithm configuration framework
Journal of Artificial Intelligence Research
Structural abstraction of software verification conditions
CAV'07 Proceedings of the 19th international conference on Computer aided verification
Improvement strategies for the F-Race algorithm: sampling design and iterative refinement
HM'07 Proceedings of the 4th international conference on Hybrid metaheuristics
A gender-based genetic algorithm for the automatic configuration of algorithms
CP'09 Proceedings of the 15th international conference on Principles and practice of constraint programming
COMPOSER: a probabilistic solution to the utility problem in speed-up learning
AAAI'92 Proceedings of the tenth national conference on Artificial intelligence
Problem structure in the presence of perturbations
AAAI'97/IAAI'97 Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligence
Time-bounded sequential parameter optimization
LION'10 Proceedings of the 4th international conference on Learning and intelligent optimization
Performance prediction and automated tuning of randomized and parametric algorithms
CP'06 Proceedings of the 12th international conference on Principles and Practice of Constraint Programming
Benchmarking SAT solvers for bounded model checking
SAT'05 Proceedings of the 8th international conference on Theory and Applications of Satisfiability Testing
Improving parallel local search for SAT
LION'05 Proceedings of the 5th international conference on Learning and Intelligent Optimization
Quantifying homogeneity of instance sets for algorithm configuration
LION'12 Proceedings of the 6th international conference on Learning and Intelligent Optimization
An analysis of post-selection in automatic configuration
Proceedings of the 15th annual conference on Genetic and evolutionary computation
Algorithm runtime prediction: Methods & evaluation
Artificial Intelligence
An analysis on separability for Memetic Computing automatic design
Information Sciences: an International Journal
Hi-index | 0.00 |
We propose an empirical analysis approach for characterizing tradeoffs between different methods for comparing a set of competing algorithm designs. Our approach can provide insight into performance variation both across candidate algorithms and across instances. It can also identify the best tradeoff between evaluating a larger number of candidate algorithm designs, performing these evaluations on a larger number of problem instances, and allocating more time to each algorithm run. We applied our approach to a study of the rich algorithm design spaces offered by three highly-parameterized, state-of-the-art algorithms for satisfiability and mixed integer programming, considering six different distributions of problem instances. We demonstrate that the resulting algorithm design scenarios differ in many ways, with important consequences for both automatic and manual algorithm design. We expect that both our methods and our findings will lead to tangible improvements in algorithm design methods.