Comparing the Effectiveness of Software Testing Strategies
IEEE Transactions on Software Engineering
Assessing Software Review Meetings: Results of a Comparative Analysis of Two Experimental Studies
IEEE Transactions on Software Engineering
Comparing and combining software defect detection techniques: a replicated empirical study
ESEC '97/FSE-5 Proceedings of the 6th European SOFTWARE ENGINEERING conference held jointly with the 5th ACM SIGSOFT international symposium on Foundations of software engineering
Applying meta-analytical procedures to software engineering experiments
Journal of Systems and Software
An Empirical Evaluation of Three Defect-Detection Techniques
Proceedings of the 5th European Software Engineering Conference
Replicated studies: building a body of knowledge about software reading techniques
Lecture notes on empirical software engineering
(Quasi-)experimental studies in industrial settings
Lecture notes on empirical software engineering
Developing techniques for using software documents: a series of empirical studies
Developing techniques for using software documents: a series of empirical studies
Reviewing 25 Years of Testing Technique Experiments
Empirical Software Engineering
A Survey of Controlled Experiments in Software Engineering
IEEE Transactions on Software Engineering
What Do We Know about Defect Detection Methods?
IEEE Software
Analysis of the influence of communication between researchers on experiment replication
Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering
The role of replications in Empirical Software Engineering
Empirical Software Engineering
On the difficulty of replicating human subjects studies in software engineering
Proceedings of the 30th international conference on Software engineering
A Framework for Software Engineering Experimental Replications
ICECCS '08 Proceedings of the 13th IEEE International Conference on on Engineering of Complex Computer Systems
Replications types in experimental disciplines
Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement
Metrics for BPEL process context-independency analysis
Service Oriented Computing and Applications
A process for managing interaction between experimenters to get useful similar replications
Information and Software Technology
Can plans and situated actions be replicated?
Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing
| Hi-index | 0.00 |
In no science or engineering discipline does it make sense to speak of isolated experiments. The results of a single experiment cannot be viewed as representative of the underlying reality. Experiment replication is the repetition of an experiment to double-check its results. Multiple replications of an experiment increase the confidence in its results. Software engineering has tried its hand at the identical (exact) replication of experiments in the way of the natural sciences (physics, chemistry, etc.). After numerous attempts over the years, apart from experiments replicated by the same researchers at the same site, no exact replications have yet been achieved. One key reason for this is the complexity of the software development setting, which prevents the many experimental conditions from being identically reproduced. This paper reports research into whether non-exact replications can be of any use. We propose a process aimed at researchers running non-exact replications. Researchers enacting this process will be able to identify new variables that are possibly having an effect on experiment results. The process consists of four phases: replication definition and planning, replication operation and analysis, replication interpretation, and analysis of the replication's contribution. To test the effectiveness of the proposed process, we have conducted a multiple-case study, revealing the variables learned from two different replications of an experiment.