Prevalence of coincidental correctness and mitigation of its impact on fault localization

  • Authors:

  • Wes Masri;Rawad Abou Assi

  • Affiliations:

  • American University of Beirut, Beirut, Lebanon;American University of Beirut, Beirut, Lebanon

  • Venue:
  • ACM Transactions on Software Engineering and Methodology (TOSEM)
  • Year:
  • 2014

Quantified Score

Hi-index 0.00

Visualization

Abstract

Researchers have argued that for failure to be observed the following three conditions must be met: CR = the defect was reached; CI = the program has transitioned into an infectious state; and CP = the infection has propagated to the output. Coincidental Correctness (CC) arises when the program produces the correct output while condition CR is met but not CP. We recognize two forms of coincidental correctness, weak and strong. In weak CC, CR is met, whereas CI might or might not be met, whereas in strong CC, both CR and CI are met. In this work we first show that CC is prevalent in both of its forms and demonstrate that it is a safety reducing factor for Coverage-Based Fault Localization (CBFL). We then propose two techniques for cleansing test suites from coincidental correctness to enhance CBFL, given that the test cases have already been classified as failing or passing. We evaluated the effectiveness of our techniques by empirically quantifying their accuracy in identifying weak CC tests. The results were promising, for example, the better performing technique, using 105 test suites and statement coverage, exhibited 9% false negatives, 30% false positives, and no false negatives nor false positives in 14.3% of the test suites. Also using 73 test suites and more complex coverage, the numbers were 12%, 19%, and 15%, respectively.