Comparing the Effectiveness of Software Testing Strategies
IEEE Transactions on Software Engineering
Software testing techniques (2nd ed.)
Software testing techniques (2nd ed.)
Assessing Software Review Meetings: Results of a Comparative Analysis of Two Experimental Studies
IEEE Transactions on Software Engineering
Comparing and combining software defect detection techniques: a replicated empirical study
ESEC '97/FSE-5 Proceedings of the 6th European SOFTWARE ENGINEERING conference held jointly with the 5th ACM SIGSOFT international symposium on Foundations of software engineering
Applying meta-analytical procedures to software engineering experiments
Journal of Systems and Software
Structured Programming; Theory and Practice the Systems Programming Series
Structured Programming; Theory and Practice the Systems Programming Series
An Empirical Evaluation of Three Defect-Detection Techniques
Proceedings of the 5th European Software Engineering Conference
Replicated studies: building a body of knowledge about software reading techniques
Lecture notes on empirical software engineering
(Quasi-)experimental studies in industrial settings
Lecture notes on empirical software engineering
Reviewing 25 Years of Testing Technique Experiments
Empirical Software Engineering
Knowledge-Sharing Issues in Experimental Software Engineering
Empirical Software Engineering
The Art of Software Testing
A Survey of Controlled Experiments in Software Engineering
IEEE Transactions on Software Engineering
What Do We Know about Defect Detection Methods?
IEEE Software
Analysis of the influence of communication between researchers on experiment replication
Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering
On the difficulty of replicating human subjects studies in software engineering
Proceedings of the 30th international conference on Software engineering
Replications types in experimental disciplines
Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement
The role of non-exact replications in software engineering experiments
Empirical Software Engineering
ICST '12 Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation
Hi-index | 0.00 |
Context: A replication is the repetition of an experiment. Several efforts have been made to adopt replication as a common practice in software engineering. There are different types of replications, depending on their purpose. Similar replications keep the experimental conditions as alike as possible to the original ones. External similar replications, where the replicating experimenters are not the same people as the original experimenters, have been a stumbling block. Several attempts at combining the results of replications have resulted in failure. Software engineering does not appear to be well suited to such replications, because it works with complex experimentally immature contexts. Software engineering settings have a large number of variables, and the role that many of them play is unknown. A successful (or useful) similar replication helps to better understand the phenomenon under study by verifying results and/or identifying contextual variables that could influence (or not) the results, through the combination of experimental results. Objective: To be able to get successful similar replications, there needs to be interaction between original and replicating experimenters. In this paper, we propose an interaction process for achieving successful similar replications. Method: This process consists of: an adaptation meeting, where experimenters tailor the experiment to the new setting; querying, to settle occasional inquiries while the experiment is being run; and a combination meeting, where experimenters meet to discuss the combination of replication outcomes with previous results. To check its effectiveness, the process has been tested on three different replications of the same experiment. Results: The proposed interaction process has helped to identify new contextual variables that could potentially influence (or not) the experimental results in the three replications run. Additionally, the interaction process has helped to uncover certain problems and deviations that occurred during some of the replications that we would have not been aware of otherwise. Conclusions: There are signs that suggest that it is possible to get successful similar replications in software engineering experimentation, when there is appropriate interaction among experimenters.