A Critical Analysis of PSP Data Quality: Results from aCase Study

  • Authors:

  • Philip M. Johnson;Anne M. Disney

  • Affiliations:

  • Dept. of Information and Computer Sciences, University of Hawaii, Honolulu, HI 96822 USA;Dept. of Information and Computer Sciences, University of Hawaii, Honolulu, HI 96822 USA

  • Venue:
  • Empirical Software Engineering
  • Year:
  • 1999

Quantified Score

Hi-index 0.00

Visualization

Abstract

The Personal SoftwareProcess (PSP) is used by software engineers to gather and analyzedata about their work. Published studies typically use data collectedusing the PSP to draw quantitative conclusions about its impactupon programmer behavior and product quality. However, our experienceusing PSP led us to question the quality of data both duringcollection and its later analysis. We hypothesized that dataquality problems can make a significant impact upon the valueof PSP measures—significant enough to lead to incorrectconclusions regarding process improvement. To test this hypothesis,we built a tool to automate the PSP and then examined 89 projectscompleted by ten subjects using the PSP manually in an educationalsetting. We discovered 1539 primary errors and categorized themby type, subtype, severity, and age. To examine the collectionproblem we looked at the 90 errors that represented impossiblecombinations of data and at other less concrete anomalies inTime Recording Logs and Defect Recording Logs. To examine theanalysis problem we developed a rule set, corrected the errorsas far as possible, and compared the original and corrected data.We found significant differences for measures such as yield andthe cost-performance ratio, confirming our hypothesis. Our resultsraise questions about the accuracy of manually collected andanalyzed PSP data, indicate that integrated tool support maybe required for high quality PSP data analysis, and suggest thatexternal measures should be used when attempting to evaluatethe impact of the PSP upon programmer behavior and product quality.