Implementation effort and performance: a comparison of custom and out-of-the-box metaheuristics on the vehicle routing problem with stochastic demand

• Authors: Paola Pellegrini;Mauro Birattari
• Venue: SLS'07 Proceedings of the 2007 international conference on Engineering stochastic local search algorithms: designing, implementing and analyzing effective heuristics
• Year: 2007

In practical applications, one can take advantage of metaheuristics in different ways: To simplify, we can say that metaheuristics can be either used out-of-the-box or a custom version can be developed. The former way requires a rather low effort, and in general allows to obtain fairly good results. The latter implies a larger investment in the design, implementation, and fine-tuning, and can often produce state-of-the-art results. Unfortunately, most of the research works proposing an empirical analysis of metaheuristics do not even try to quantify the development effort devoted to the algorithms under consideration. In other words, they do not make clear whether they considered out-of-the-box or custom implementations of the metaheuristics under analysis. The lack of this information seriously undermines the generality and utility of these works. The aim of the paper is to stress that results obtained with out-of-the-box implementations cannot be always generalized to custom ones, and vice versa. As a case study, we focus on the vehicle routing problem with stochastic demand and on five among the most successful metaheuristics--namely, tabu search, simulated annealing, genetic algorithm, iterated local search, and ant colony optimization. We show that the relative performance of these algorithms strongly varies whether one considers out-of-the-box implementations or custom ones, in which the parameters are accurately fine-tuned.