On the complexity of blocks-world planning
Artificial Intelligence
Generating hard satisfiability problems
Artificial Intelligence - Special volume on frontiers in problem solving: phase transitions and complexity
Proceedings of the 1997 international symposium on Physical design
A Computing Procedure for Quantification Theory
Journal of the ACM (JACM)
A machine program for theorem-proving
Communications of the ACM
Chaff: engineering an efficient SAT solver
Proceedings of the 38th annual Design Automation Conference
SATIRE: a new incremental satisfiability engine
Proceedings of the 38th annual Design Automation Conference
Local Search Algorithms for SAT: An Empirical Evaluation
Journal of Automated Reasoning
Design of Experiments for Evaluation of BDD Packages Using Controlled Circuit Mutations
FMCAD '98 Proceedings of the Second International Conference on Formal Methods in Computer-Aided Design
SATO: An Efficient Propositional Prover
CADE-14 Proceedings of the 14th International Conference on Automated Deduction
Generation of tightly controlled equivalence classes for experimental design of heuristics for graph-based np-hard problems
Heuristics, Experimental Subjects, and Treatment Evaluation in Bigraph Crossing Minimization
Journal of Experimental Algorithmics (JEA)
Evidence for invariants in local search
AAAI'97/IAAI'97 Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligence
Evaluating las vegas algorithms: pitfalls and remedies
UAI'98 Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence
Effective bounding techniques for solving unate and binate covering problems
Proceedings of the 42nd annual Design Automation Conference
High-contrast algorithm behavior: observation, hypothesis, and experimental design
Proceedings of the 2007 workshop on Experimental computer science
Performance testing of combinatorial solvers with isomorph class instances
Proceedings of the 2007 workshop on Experimental computer science
High-contrast algorithm behavior: observation, conjecture, and experimental design
ecs'07 Experimental computer science on Experimental computer science
Performance testing of combinatorial solvers with isomorph class instances
ecs'07 Experimental computer science on Experimental computer science
Benchmarking in digital circuit design
MINO'08 Proceedings of the 7th WSEAS International Conference on Microelectronics, Nanoelectronics, Optoelectronics
Experimenting with Small Changes in Conflict-Driven Clause Learning Algorithms
CP '08 Proceedings of the 14th international conference on Principles and Practice of Constraint Programming
Benchmarking in digital circuit design automation
WSEAS Transactions on Circuits and Systems
Careful ranking of multiple solvers with timeouts and ties
SAT'11 Proceedings of the 14th international conference on Theory and application of satisfiability testing
Performance prediction and automated tuning of randomized and parametric algorithms
CP'06 Proceedings of the 12th international conference on Principles and Practice of Constraint Programming
Statistical methodology for comparison of SAT solvers
SAT'10 Proceedings of the 13th international conference on Theory and Applications of Satisfiability Testing
A survey of the satisfiability-problems solving algorithms
International Journal of Advanced Intelligence Paradigms
Hi-index | 0.00 |
A recent series of experiments with a group of state-of-the-art SAT solvers and several well-defined classes of problem instances reports statistically significant performance variability for the solvers. A systematic analysis of the observed performance data, all openly archived on the Web, reveals distributions which we classify into three broad categories: (1) readily characterized with a simple χ2-test, (2) requiring more in-depth analysis by a statistician, (3) incomplete, due to time-out limit reached by specific solvers. The first category includes two well-known distributions: normal and exponential; we use simple first-order criteria to decide the second category and label the distributions as near-normal, near-exponential and heavy-tail. We expect that good models for some if not most of these may be found with parameters that fit either generalized gamma, Weibull, or Pareto distributions. Our experiments show that most SAT solvers exhibit either normal or exponential distribution of execution time (runtime) on many equivalence classes of problem instances. This finding suggests that the basic mathematical framework for these experiments may well be the same as the one used to test the reliability or lifetime of hardware components such as lightbulbs, A/C units, etc. A batch of N replicated hardware components represents an equivalence class of N problem instances in SAT, a controlled operating environment A represents a SAT solver A, and the survival function RA(x) (where x represents the lifetime) is the complement of the solvability function SA(x)=1−RA(x) where x may represent runtime, implications, backtracks, etc. As demonstrated in the paper, a set of unrelated benchmarks or randomly generated SAT instances available today cannot measure the performance of SAT solvers reliably – there is no control on their ‘hardness’. However, equivalence class instances as defined in this paper are, in effect, replicated instances of a specific reference instance. The proposed method not only provides a common platform for a systematic study and a reliable improvement of deterministic and stochastic SAT solvers alike but also supports the introduction and validation of new problem instance classes.