The impact of limited search procedures for systematic literature reviews A participant-observer case study

  • Authors:

  • Barbara Kitchenham;Pearl Brereton;Mark Turner;Mahmood Niazi;Stephen Linkman;Rialette Pretorius;David Budgen

  • Affiliations:

  • School of Computing and Mathematics, Keele University, Stoke-on-Trent ST5 5BG, UK;School of Computing and Mathematics, Keele University, Stoke-on-Trent ST5 5BG, UK;School of Computing and Mathematics, Keele University, Stoke-on-Trent ST5 5BG, UK;School of Computing and Mathematics, Keele University, Stoke-on-Trent ST5 5BG, UK;School of Computing and Mathematics, Keele University, Stoke-on-Trent ST5 5BG, UK;Dept Computer Science, Durham University, Durham, DH1 3LE, UK;Dept Computer Science, Durham University, Durham, DH1 3LE, UK

  • Venue:
  • ESEM '09 Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

This study aims to compare the use of targeted manual searches with broad automated searches, and to assess the importance of grey literature and breadth of search on the outcomes of SLRs. We used a participant-observer multi-case embedded case study. Our two cases were a tertiary study of systematic literature reviews published between January 2004 and June 2007 based on a manual search of selected journals and conferences and a replication of that study based on a broad automated search. Broad searches find more papers than restricted searches, but the papers may be of poor quality. Researchers undertaking SLRs may be justified in using targeted manual searches if they intend to omit low quality papers; if publication bias is not an issue; or if they are assessing research trends in research methodologies.