Comparing Detection Methods For Software Requirements Inspections: A Replication Using Professional Subjects

• Authors: Adam Porter;Lawrence Votta
• Venue: Empirical Software Engineering
• Year: 1998

Software requirementsspecifications (SRS) are often validated manually. One such processis inspection, in which several reviewers independently analyzeall or part of the specification and search for faults. Thesefaults are then collected at a meeting of the reviewers and author(s). Usually, reviewers use Ad Hoc or Checklistmethods to uncover faults. These methods force all reviewersto rely on nonsystematic techniques to search for a wide varietyof faults. We hypothesize that a Scenario-based method, in whicheach reviewer uses different, systematic techniques to searchfor different, specific classes of faults, will have a significantlyhigher success rate. In previous work we evaluatedthis hypothesis using 48 graduate students in computer scienceas subjects. We now have replicated this experimentusing 18 professional developers from Lucent Technologies assubjects. Our goals were to (1) extend the external credibilityof our results by studying professional developers, and to (2)compare the performances of professionals with that of the graduatestudents to better understand how generalizable the results ofthe less expensive student experiments were. For each inspection we performed four measurements: (1) individualfault detection rate, (2) team fault detection rate, (3) percentageof faults first identified at the collection meeting (meetinggain rate), and (4) percentage of faults first identified byan individual, but never reported at the collection meeting (meetingloss rate). For both the professionals and thestudents the experimental results are that (1) the Scenario methodhad a higher fault detection rate than either Ad Hoc or Checklistmethods, (2) Checklist reviewers were no more effective thanAd Hoc reviewers, (3) Collection meetings produced no net improvementin the fault, and detection rate—meeting gains were offsetby meeting losses, Finally, although specificmeasures differed between the professional and student populations,the outcomes of almost all statistical tests were identical.This suggests that the graduate students provided an adequatemodel of the professional population and that the much greaterexpense of conducting studies with professionals may not alwaysbe required.