Evaluation studies of software testing research in Brazil and in the world: A survey of two premier software engineering conferences

  • Authors:

  • OtáVio Augusto Lazzarini Lemos;Fabiano Cutigi Ferrari;Marcelo Medeiros Eler;José Carlos Maldonado;Paulo Cesar Masiero

  • Affiliations:

  • Departamento de Ciência e Tecnologia, Universidade Federal de São Paulo (UNIFESP), São José dos Campos, SP, Brazil;Departamento de Computação, Universidade Federal de São Carlos (UFSCar), São Carlos, SP, Brazil;Departamento de Sistemas de Computação, Universidade de São Paulo (ICMC/USP), São Carlos, SP, Brazil;Departamento de Sistemas de Computação, Universidade de São Paulo (ICMC/USP), São Carlos, SP, Brazil;Departamento de Sistemas de Computação, Universidade de São Paulo (ICMC/USP), São Carlos, SP, Brazil

  • Venue:
  • Journal of Systems and Software
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper reports on a historical perspective of the evaluation studies present in software testing research published in the Brazilian Symposium on Software Engineering (SBES) in comparison to the International Conference on Software Engineering (ICSE). The survey characterizes the software testing-related papers published in the 25-year history of SBES, investigates the types of evaluation presented in these publications, and how the rate of evaluations has evolved over the years. A similar analysis within the same period is made for ICSE, allowing for a comparison between the national and international scenario. Results show that the rate of papers that present evaluation studies in SBES has significantly increased over the years. However, among the papers that described some kind of evaluation, only around 20% performed more rigorous evaluations (i.e. case studies, quasi experiments, or controlled experiments). Such percentage is low when compared to ICSE, which presented 40% of papers with more rigorous evaluations within the same period. Nevertheless, we noticed that both venues still lack the publication of research reporting controlled experiments: only a single paper in each conference presented this type of evaluation.