An ecological perspective towards the evolution of quantitative studies in software engineering

• Authors: Rafael M. de Mello;Guilherme H. Travassos
• Venue: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering
• Year: 2013

Context: Two of the most common external threats to validity in quantitative studies in software engineering (SE) are concerned with defining the population by convenience and nonrandom sampling assignment. Although these limitations can be reduced by increasing the number of replications and aggregating their results, the acquired evidence rarely can be generalized to the field. Objective: To investigate the state of practice of meta-analysis in SE and its limitations, intending to propose an alternative perspective to understand the relationships among experimentation, production, threats to validity and evidence. To propose and evaluate means to strengthen quantitative studies in software engineering and making them less risky due to population and sampling issues. Method: To use the underlying idea from the Theory of Food Chains to alternatively understand the impact of external threats to validity in the SE experimental cycle (experimental chains). Next, to accomplish an initial technical literature survey to observe basic features of secondary studies aggregating primary studies results. Third, to organize a set of experimental chain's concepts and make initial discussions regarding the observed secondary studies concerned with this metaphor. Results: By applying the experimental chains concepts it was initially observed that, although important and necessary, most of the current effort in the conduction of quantitative studies in SE does not produce (mainly due to population/sampling constraints) results strong enough to positively impact the engineering of software. It promotes an imbalance between research and practice. However, more investigation is necessary to support this claim. Conclusion: We argue that research energy has been lost in SE studies due to population/sampling constraints. Therefore, we believe more investigation must be undertaken to understand how better organizing, enlarging, setting up and sampling SE quantitative studies' population by using, for instance, alternative technologies such as social networks or other crowdsourcing technologies.