On the Repeatability of Metric Models and Metrics across Software Builds

• Authors: Norman F. Schneidewind
• Venue: ISSRE '00 Proceedings of the 11th International Symposium on Software Reliability Engineering
• Year: 2000

Over the past several years we have developed the following metrics models: Boolean discriminate functions (BDFs) for classifying quality; Kolmogorov-Smirnov distance for estimating metric critical values; various derivative calculations for assessing the quality that could be achieved with various levels of quality control and inspection; a stopping rule for deciding how many metrics to use in a discriminate function; point and confidence interval estimates of quality; Relative Critical Value Deviation metrics for indexing quality; and non-linear regression functions for predicting quality. We would like these models and metrics to be repeatable across the n builds of a software system. The great advantage of repeatability is that models and metrics only need to be developed and validated once on build 1 and then applied n-1 times without modification to subsequent builds, with considerable savings in analysis and computational effort. In practical terms, this approach involves using the same model parameters (e.g., metrics critical values) that were validated and applying them unchanged on subsequent builds. The disadvantage is that the quality and metrics data of builds 2... n, which varies across builds, is not utilized. We make a comparison of this n approach with one that involves validating models and metrics on each build i and applying them only on build i+1, and then repeating the process. The advantage of this approach is that all available data are used in the models and analysis but at considerable cost in effort. We report on experiments involving large sets of discrepancy report and metrics data on the Space Shuttle flight software, where we compare the predictive accuracy and effort of the two approaches for BDFs, critical values, derivative quality and inspection calculations, and stopping rule.