Statistical Methods for Analyzing Speedup Learning Experiments

Authors:
Oren Etzioni;Ruth Etzioni
Affiliations:
Department of Computer Science and Engineering, FR-35, University of Washington, Seattle, WA 98195. ETZIONI@CS.WASHINGTON.EDU;Fred Hutchinson Cancer Research Center, Division of Public Health Sciences, Seattle, WA 98104 and Department of Biostatistics, University of Washington, Seattle, WA 98195
Venue:
Machine Learning
Year:
1994

Citing 0
Cited 11

Speedup learning for repair-based search by identifying redundant steps

The Journal of Machine Learning Research
Comparing information‐theoretic attribute selection measures: a statistical approach

AI Communications
Building and refining abstract planning cases by change of representation language

Journal of Artificial Intelligence Research
Adaptive problem-solving for large-scale scheduling problems: a case study

Journal of Artificial Intelligence Research
A selective macro-learning algorithm and its application to the N × N sliding-tile puzzle

Journal of Artificial Intelligence Research
The COMPSET algorithm for subset selection

IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Integrating heuristics for constraint satisfaction problems: a case study

AAAI'93 Proceedings of the eleventh national conference on Artificial intelligence
Relative utility of EBG based plan reuse in partial ordering vs. total ordering planning

AAAI'93 Proceedings of the eleventh national conference on Artificial intelligence
Building Mashups by Demonstration

ACM Transactions on the Web (TWEB)
Tradeoffs in the empirical evaluation of competing algorithm designs

Annals of Mathematics and Artificial Intelligence
Statistical methodology for comparison of SAT solvers

SAT'10 Proceedings of the 13th international conference on Theory and Applications of Satisfiability Testing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Speedup learning systems are typically evaluated by comparing their impact on a problem solver's performance. The impact is measured by running the problem solver, before and after learning, on a sample of problems randomly drawn from some distribution. Often, the experimenter imposes a bound on the CPU time the problem solver is allowed to spend on any individual problem. Segre et al. (1991) argue that the experimenter's choice of time bound can bias the results of the experiment. To address this problem, we present statistical hypothesis tests specifically designed to analyze speedup data and eliminate this bias. We apply the tests to the data reported by Etzioni (1990a) and show that most (but not all) of the speedups observed are statistically significant.