A Formal Process for Evaluating COTS Software Products

  • Authors:

  • Patricia K. Lawlis;Kathryn E. Mark;Deborah A. Thomas;Terry Courtheyn

  • Affiliations:

  • -;-;-;-

  • Venue:
  • Computer
  • Year:
  • 2001

Quantified Score

Hi-index 4.10

Visualization

Abstract

A software-product evaluation process grounded in mathematics and decision theory can effectively determine product quality and suitability with less risk and at lower cost than conventional methods. Thus, the authors seek to develop a formal process for effective software-product evaluation to assess commercial-off-the-shelf product quality and suitability before purchase. They propose a requirements-based COTS product evaluation process, RCPEP, that ensures a quality outcome, then demonstrate its use in a case study of a US Air Force product for a large, component-based training-management system. The authors describe how, unlike a typical ad hoc product evaluation, RCPEP relies on user-defined requirements to determine product suitability and quality. RCPEP identifies every product that addresses the requirements, conducts a trade study to narrow the list to serious candidates, and evaluates the remaining candidates using hands-on scenarios. An ad hoc process does not adhere to strict controls that ensure all candidate products receive identical evaluations and generate comparable results. In contrast, RCPEP's hands-on evaluation includes several controls that ensure the evaluation does not compromise the results' validity. The Air Force case study confirmed the authors' expectation that the simplest form of decision analysis, weighted averages, could effectively evaluate COTS products. RCPEP succeeded because it relied on user-defined requirements, contained a sufficient number of evaluators to obtain valid quantitative results, and adhered to the controls. The authors believe that RCPEP will be effective for all forms of COTS product evaluations and could mature into an industry standard.