Practical experiments are informative, but never perfect

  • Authors:

  • Rosalva E. Gallardo-Valencia;Vivian Olivera;Susan Elliott Sim

  • Affiliations:

  • University of California, Irvine, Irvine, CA;University of California, Irvine, Irvine, CA;University of California, Irvine, Irvine, CA

  • Venue:
  • Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

The design of empirical experiments involves making design decisions to trade off what is ideal against what is achievable. Researchers must weigh limitations on resources, metrics, and the current state of knowledge, against the validity of the results. In this paper, we report on the design decisions we made in a small controlled experiment and their effects on the conclusions of the study. The goal of the study was to measure the impact of requirements formats on maintenance tasks. We encountered problems with the subjects' lack of expertise in the technology used, the equivalence of subjects in our experiment conditions, and the number of subjects. These issues meant that we were able to draw conclusions about how subjects worked with the requirements formats, but not about the effect of the formats on the completeness of the implementation. We had a practical and doable experiment, but our results were not conclusive, only informative.