(Quasi-)experimental studies in industrial settings
Lecture notes on empirical software engineering
The Case Against Cross-Over Designs in Software Engineering
STEP '03 Proceedings of the Eleventh Annual International Workshop on Software Technology and Engineering Practice
A Survey of Controlled Experiments in Software Engineering
IEEE Transactions on Software Engineering
A framework for the analysis of software cost estimation accuracy
Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering
The Future of Empirical Methods in Software Engineering Research
FOSE '07 2007 Future of Software Engineering
A Systematic Review of Theory Use in Software Engineering Experiments
IEEE Transactions on Software Engineering
Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise
IEEE Transactions on Software Engineering
Systematic review: A systematic review of effect size in software engineering experiments
Information and Software Technology
A systematic review of quasi-experiments in software engineering
Information and Software Technology
Empirical paradigm - the role of experiments
Proceedings of the 2006 international conference on Empirical software engineering issues: critical assessment and future directions
Strength of evidence in systematic reviews in software engineering
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement
A systematic review of quasi-experiments in software engineering
Information and Software Technology
How to compare program comprehension in FOSD empirically: an experience report
FOSD '09 Proceedings of the First International Workshop on Feature-Oriented Software Development
Information and Software Technology
A systematic review of evaluation of variability management approaches in software product lines
Information and Software Technology
Using mapping studies as the basis for further research - A participant-observer case study
Information and Software Technology
Identifying relevant studies in software engineering
Information and Software Technology
Six years of systematic literature reviews in software engineering: An updated tertiary study
Information and Software Technology
Understanding technology use in global virtual teams: Research methodologies and methods
Information and Software Technology
Design and evaluation of a process for identifying architecture patterns in open source software
ECSA'11 Proceedings of the 5th European conference on Software architecture
A status report on the evaluation of variability management approaches
EASE'09 Proceedings of the 13th international conference on Evaluation and Assessment in Software Engineering
The value of mapping studies: a participantobserver case study
EASE'10 Proceedings of the 14th international conference on Evaluation and Assessment in Software Engineering
Flexibility in research designs in empirical software engineering
EASE'08 Proceedings of the 12th international conference on Evaluation and Assessment in Software Engineering
ICWE'12 Proceedings of the 12th international conference on Web Engineering
Questioning software maintenance metrics: a comparative case study
Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement
MDD vs. traditional software development: A practitioner's subjective perspective
Information and Software Technology
An ecological perspective towards the evolution of quantitative studies in software engineering
Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering
Distributed debugging for mobile networks
Journal of Systems and Software
Hi-index | 0.00 |
Background:: Experiments in which study units are assigned to experimental groups nonrandomly are called quasi-experiments. They allow investigations of cause-effect relations in settings in which randomization is inappropriate, impractical, or too costly. Problem outline:: The procedure by which the nonrandom assignments are made might result in selection bias and other related internal validity problems. Selection bias is a systematic (not happening by chance) pre-experimental difference between the groups that could influence the results. By detecting the cause of the selection bias, and designing and analyzing the experiments accordingly, the effect of the bias may be reduced or eliminated. Research method:: To investigate how quasi-experiments are performed in software engineering (SE), we conducted a systematic review of the experiments published in nine major SE journals and three conference proceedings in the decade 1993-2002. Results:: Among the 113 experiments detected, 35% were quasi-experiments. In addition to field experiments, we found several applications for quasi-experiments in SE. However, there seems to be little awareness of the precise nature of quasi-experiments and the potential for selection bias in them. The term ''quasi-experiment'' was used in only 10% of the articles reporting quasi-experiments; only half of the quasi-experiments measured a pretest score to control for selection bias, and only 8% reported a threat of selection bias. On average, larger effect sizes were seen in randomized than in quasi-experiments, which might be due to selection bias in the quasi-experiments. Conclusion:: We conclude that quasi-experimentation is useful in many settings in SE, but their design and analysis must be improved (in ways described in this paper), to ensure that inferences made from this kind of experiment are valid.