Improving scenario testing process by adding value-based prioritization: an industrial case study

Authors:
Qi Li;Barry Boehm
Affiliations:
University of Southern California, USA;University of Southern California, USA
Venue:
Proceedings of the 2013 International Conference on Software and System Process
Year:
2013

Citing 29
Cited 0

First things first

Software Development
Prioritizing test cases for regression testing

Proceedings of the 2000 ACM SIGSOFT international symposium on Software testing and analysis
The prediction of faulty classes using object-oriented design metrics

Journal of Systems and Software
Incorporating varying test costs and fault severities into test case prioritization

ICSE '01 Proceedings of the 23rd International Conference on Software Engineering
Prioritizing Test Cases For Regression Testing

IEEE Transactions on Software Engineering
Test Case Prioritization: A Family of Empirical Studies

IEEE Transactions on Software Engineering
Risk Based E-Business Testing

Risk Based E-Business Testing
Effectively prioritizing tests in development environment

ISSTA '02 Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis
Operational Profiles in Software-Reliability Engineering

IEEE Software
Value-based software engineering

ACM SIGSOFT Software Engineering Notes
Test Case Prioritization: An Empirical Study

ICSM '99 Proceedings of the IEEE International Conference on Software Maintenance
Selecting a Cost-Effective Test Case Prioritization Technique

Software Quality Control
Establishment of Automated Regression Testing at ABB: Industrial Experience Report on 'Avoiding the Pitfalls'

Proceedings of the 19th IEEE international conference on Automated software engineering
Predicting the Location and Number of Faults in Large Software Systems

IEEE Transactions on Software Engineering
Mining metrics to predict component failures

Proceedings of the 28th international conference on Software engineering
How Much Software Quality Investment Is Enough: A Value-Based Approach

IEEE Software
An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models

Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering
Automating algorithms for the identification of fault-prone files

Proceedings of the 2007 international symposium on Software testing and analysis
Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research (Practitioners)

Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research (Practitioners)
Predicting defects using network analysis on dependency graphs

Proceedings of the 30th international conference on Software engineering
Using sensitivity analysis to create simplified economic models for regression testing

ISSTA '08 Proceedings of the 2008 international symposium on Software testing and analysis
Do Crosscutting Concerns Cause Defects?

IEEE Transactions on Software Engineering
Can developer-module networks predict failures?

Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering
An empirical study of the effect of time constraints on the cost-benefits of regression testing

Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering
Time-aware test-case prioritization using integer linear programming

Proceedings of the eighteenth international symposium on Software testing and analysis
Putting It All Together: Using Socio-technical Networks to Predict Failures

ISSRE '09 Proceedings of the 2009 20th International Symposium on Software Reliability Engineering
A value-based review process for prioritizing artifacts

Proceedings of the 2011 International Conference on Software and Systems Process
CRANE: Failure Prediction, Change Analysis and Test Prioritization in Practice -- Experiences from Windows

ICST '11 Proceedings of the 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation
Regression testing minimization, selection and prioritization: a survey

Software Testing, Verification & Reliability

Quantified Score

Hi-index	0.00

Visualization

Abstract

Most of the current testing strategies treat all aspects of software equally important in a value-neutral way; this becomes more risky when the testing resources are limited. Our goal in this case study aims at improving the testing cost-effectiveness of an industrial scenario testing process under time constraints. We proposed a value-based testing prioritization strategy which allows tests to be ranked by how well the tests can reduce risk exposure. Combining this with the tests’ relative costs enables them to be prioritized in terms of return on investment (ROI) or risk reduction leverage (RRL). Besides, a new metric Average Percentage of Business Importance Earned (APBIE) is proposed to measure how quickly the testing can reduce the quality uncertainty and earn the relative business importance of the system under test (SUT).The results from one case study to prioritize operational testing scenarios in Galorath Inc. show that our method can improve the testing cost-effectiveness in terms of APBIE.