Building Knowledge through Families of Experiments
IEEE Transactions on Software Engineering
Knowledge-Sharing Issues in Experimental Software Engineering
Empirical Software Engineering
Text mining for software engineering: how analyst feedback impacts final results
MSR '05 Proceedings of the 2005 international workshop on Mining software repositories
Analysis of the influence of communication between researchers on experiment replication
Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering
Beyond source code: the importance of other artifacts in software development (a case study)
Journal of Systems and Software - Special issue: Selected papers from the 4th source code analysis and manipulation (SCAM 2004) workshop
Predicting Eclipse Bug Lifetimes
MSR '07 Proceedings of the Fourth International Workshop on Mining Software Repositories
The role of replications in Empirical Software Engineering
Empirical Software Engineering
Guest Editors' Introduction: Reproducible Research
Computing in Science and Engineering
Reproducible Research in Computational Harmonic Analysis
Computing in Science and Engineering
FLOSSMetrics: Free/Libre/Open Source Software Metrics
CSMR '09 Proceedings of the 2009 European Conference on Software Maintenance and Reengineering
Macro-level software evolution: a case study of a large software compilation
Empirical Software Engineering
Replicating software engineering experiments: a poisoned chalice or the Holy Grail
Information and Software Technology
1st International Workshop on Replication in Empirical Software Engineering Research (RESER)
Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2
Proceedings of the FSE/SDP workshop on Future of software engineering research
Special issue on repeatable results in software engineering prediction
Empirical Software Engineering
Boa: a language and infrastructure for analyzing ultra-large-scale software repositories
Proceedings of the 2013 International Conference on Software Engineering
Proceedings of the 10th Working Conference on Mining Software Repositories
A Data Restore Model for Reproducibility in Computational Statistics
Proceedings of the 13th International Conference on Knowledge Management and Knowledge Technologies
Preliminary lessons from a software evolution analysis of Moodle
Proceedings of the First International Conference on Technological Ecosystem for Enhancing Multiculturality
ACM Computing Surveys (CSUR)
Hi-index | 0.00 |
Among empirical software engineering studies, those based on data retrieved from development repositories (such as those of source code management, issue tracking or communication systems) are specially suitable for reproduction. However their reproducibility status can vary a lot, from easy to almost impossible to reproduce. This paper explores which elements can be considered to characterize the reproducibility of a study in this area, and how they can be analyzed to better understand the type of reproduction studies they enable or obstruct. One of the main results of this exploration is the need of a systematic approach to asses the reproducibility of a study, due to the complexity of the processes usually involved, and the many details to be taken into account. To address this need, a methodology for assessing the reproducibility of studies is also presented and discussed, as a tool to help to raise awareness about research reproducibility in this field. The application of the methodology in practice has shown how, even for papers aimed to be reproducible, a systematic analysis raises important aspects that render reproduction difficult or impossible. We also show how, by identifying elements and attributes related to reproducibility, it can be better understood which kind of reproduction can be done for a specific study, given the description of datasets, methodologies and parameters it uses.