A web-based support environment for software engineering experiments

  • Authors:

  • Erik Arisholm;Dag I. K. Sjøberg;Gunnar J. Carelius;Yngve Lindsjørn

  • Affiliations:

  • Simula Research Laboratory, P.O. Box 134, N0-1325 Lysaker, Norway;Simula Research Laboratory, P.O. Box 134, N0-1325 Lysaker, Norway;Simula Research Laboratory, P.O. Box 134, N0-1325 Lysaker, Norway;KompetanseWeb AS Tullinsgate 6, NO-0166 Oslo, Norway

  • Venue:
  • Nordic Journal of Computing
  • Year:
  • 2002

Quantified Score

Hi-index 0.02

Visualization

Abstract

The software engineering communities frequently propose new software engineering technologies, such as new development techniques, programming languages and tools, without rigorous scientific evaluation. One way to evaluate software engineering technologies is through controlled experiments where the effects of the technology can be isolated from confounding factors, i.e., establishing cause-effect relationships. For practical and financial reasons, however, such experiments are often quite unrealistic, typically involving students in a class-room environment solving small pen-and-paper tasks. A common criticism of the results of the experiments is their lack of external validity, i.e., that the results are not valid outside the experimental conditions. To increase the external validity of the experimental results, the experiments need to be more realistic. The realism can be increased using professional developers as subjects who conduct larger experimental tasks in their normal work environment. However, the logistics involved in running such experiments are tremendous. More specifically, the experimental materials (e.g., questionnaires, task descriptions, code and tools) must be distributed to each programmer, the progress of the experiment needs to be controlled and monitored, and the results of the experiment need to be collected and analyzed. To support this logistics for large-scale, controlled experiments, we have developed a web-based experiment support environment called SESE. This paper describes SESE, its development and the experiences from using it to conduct a large controlled experiment in industry.