Do bad smells indicate "trouble" in code?
DEFECTS '08 Proceedings of the 2008 workshop on Defects in large software systems
A systematic review of systematic review process research in software engineering
Information and Software Technology
| Hi-index | 0.00 |
Context: A systematic literature review is being carried out as part of a Ph. D. research project, which is focussing on the security problems caused by inference in distributed and autonomous information systems. Inference problems differ from other security problems because they do not directly violate access control mechanisms. For example, a user may infer an aircraft's secret destination using unclassified values for the range of the aircraft and its direction of flight. The aim of the research is to investigate the effects of inference when information is collated from autonomous, heterogeneous and distributed information systems (Budgen et al., 2005). Objectives: The lessons learnt while planning and conducting a systematic literature review are presented. Results: The lessons learnt are: L1: A single researcher can develop a systematic review protocol using stepwise refinement to make maximal use of the advice provided by a supervisor. This is contrasted with piloting the protocol, which is more suitable when multiple researchers can review the protocol, making fewer and more extensive changes. L2: The questions in a data extraction form do not have to be as detailed, when used by only one researcher, because they are not required to cater for the different possible interpretations of multiple researchers. L3: To write a protocol requires knowledge of the research area, and this knowledge can be used to aid the generation of search strings, and validate the systematic review search process. L4: The systematic search process highlighted sources that the original ad hoc review dismissed as irrelevant. L5: Trialling all the stages of the protocol helps develop the research beyond that of the systematic review. L6: It was helpful to read around the occurrences, in the body of a paper, of the words used in the search terms (defined in the protocol), as well as using the abstract and conclusion, to determine the relevance of a paper. L7: A terminating condition, which defines when the systematic review will end, ensures that the review does not continue beyond the time available to a research student. Conclusions: Conducting a systematic review is a time consuming process for a single researcher and could easily extend beyond the allocated time of a research project. Our experience indicates that, for Ph. D. students, there is a need for a lighter systematic review process because of the differences between groups of researchers carrying out a systematic review compared to a single researcher. One of the steps towards a lighter process, which addresses the needs of a single researcher, is the use of stepwise refinement to develop the protocol. To address the time constraints of a review, terminating conditions can be applied to the processes in the review protocol, which ensures the review does not continue beyond the time available to a research student.