Software errors and complexity: an empirical investigation0
Communications of the ACM
Identifying Error-Prone Software An Empirical Study
IEEE Transactions on Software Engineering
An Analysis of Several Software Defect Models
IEEE Transactions on Software Engineering
Reliability Analysis of Large Software Systems: Defect Data Modeling
IEEE Transactions on Software Engineering
Design metrics and aids to their automatic collection
Information and Software Technology - Software quality assurance
Prediction and control of ADA software defects
Journal of Systems and Software - An Oregon workshop on software metrics
An evaluation of some design metrics
Software Engineering Journal - Special issue: on software reliability and metrics
Software complexity: measures and methods
Software complexity: measures and methods
The cleanroom approach to quality software development
The cleanroom approach to quality software development
The Detection of Fault-Prone Programs
IEEE Transactions on Software Engineering
Managing computer projects: avoiding the pitfalls
Managing computer projects: avoiding the pitfalls
Practical software metrics for project management and process improvement
Practical software metrics for project management and process improvement
Projecting Software Defects from Analyzing Ada Designs
IEEE Transactions on Software Engineering - Special issue on software measurement principles, techniques, and environments
A Practical View of Software Measurement and Implementation Experiences Within Motorola
IEEE Transactions on Software Engineering - Special issue on software measurement principles, techniques, and environments
Estimating software fault content before coding
ICSE '92 Proceedings of the 14th international conference on Software engineering
Software metrics (2nd ed.): a rigorous and practical approach
Software metrics (2nd ed.): a rigorous and practical approach
Predicting Fault-Prone Software Modules in Telephone Switches
IEEE Transactions on Software Engineering
A Critique of Software Defect Prediction Models
IEEE Transactions on Software Engineering
Using Formal Description Techniques: An Introduction to Estelle, Lotos, and SDL
Using Formal Description Techniques: An Introduction to Estelle, Lotos, and SDL
Handbook of Software Quality Assurance
Handbook of Software Quality Assurance
Operational Profiles in Software-Reliability Engineering
IEEE Software
Static Inspection: Tapping the Wheels of Software
IEEE Software
Software Measurement: A Necessary Scientific Basis
IEEE Transactions on Software Engineering
Software Reliability Analysis of Three Successive Generations of a Switching System
EDCC-1 Proceedings of the First European Dependable Computing Conference on Dependable Computing
Reliability of a commercial telecommunications system
ISSRE '96 Proceedings of the The Seventh International Symposium on Software Reliability Engineering
Application of multivariate analysis for software fault prediction
Software Quality Control
A Critique of Software Defect Prediction Models
IEEE Transactions on Software Engineering
Proceedings of the Conference on The Future of Software Engineering
Economics of software verification
PASTE '01 Proceedings of the 2001 ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering
The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics
IEEE Transactions on Software Engineering
An empirical study of operating systems errors
SOSP '01 Proceedings of the eighteenth ACM symposium on Operating systems principles
Object-oriented metrics: A review of theory and practice
Advances in software engineering
Usage testing of military simulation systems
Proceedings of the 33nd conference on Winter simulation
The distribution of faults in a large industrial software system
ISSTA '02 Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis
The Optimal Class Size for Object-Oriented Software
IEEE Transactions on Software Engineering
An empirical evaluation of fault-proneness models
Proceedings of the 24th International Conference on Software Engineering
Quantitative Analysis of Development Defects to Guide Testing: A Case Study
Software Quality Control
Software Measurement: Uncertainty and Causal Modeling
IEEE Software
Comparing Partition and Random Testing via Majorization and Schur Functions
IEEE Transactions on Software Engineering
Probabilistic Modelling for Software Quality Control
ECSQARU '01 Proceedings of the 6th European Conference on Symbolic and Quantitative Approaches to Reasoning with Uncertainty
Identifying high maintenance legacy software
Journal of Software Maintenance: Research and Practice
Architectural level risk assessment tool based on UML specifications
Proceedings of the 25th International Conference on Software Engineering
An empirical evaluation of system and regression testing
CASCON '02 Proceedings of the 2002 conference of the Centre for Advanced Studies on Collaborative research
Using Code Metrics to Predict Maintenance of Legacy Programs: A Case Study
ICSM '01 Proceedings of the IEEE International Conference on Software Maintenance (ICSM'01)
An Objective Measure of Digital System Design Quality
ISQED '00 Proceedings of the 1st International Symposium on Quality of Electronic Design
Thresholds for Object-Oriented Measures
ISSRE '00 Proceedings of the 11th International Symposium on Software Reliability Engineering
Module Size Distribution and Defect Density
ISSRE '00 Proceedings of the 11th International Symposium on Software Reliability Engineering
Operational anomalies as a cause of safety-critical requirements evolution
Journal of Systems and Software
An empirical comparison and characterization of high defect and high complexity modules
Journal of Systems and Software
New Directions in Measurement for Software Quality Control
STEP '02 Proceedings of the 10th International Workshop on Software Technology and Engineering Practice
An Empirical Validation of Object-Oriented Metrics in Two Different Iterative Software Processes
IEEE Transactions on Software Engineering
Using operational distributions to judge testing progress
Proceedings of the 2003 ACM symposium on Applied computing
Efficient use of code coverage in large-scale software development
CASCON '03 Proceedings of the 2003 conference of the Centre for Advanced Studies on Collaborative research
An Empirical Study of Software Reuse vs. Defect-Density and Stability
Proceedings of the 26th International Conference on Software Engineering
Making Resource Decisions for Software Projects
Proceedings of the 26th International Conference on Software Engineering
ISSTA '04 Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis
Journal of Systems and Software
A Maintenance-Oriented Fault Model for the DECOS Integrated Diagnostic Architecture
IPDPS '05 Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Workshop 2 - Volume 03
Use of relative code churn measures to predict system defect density
Proceedings of the 27th international conference on Software engineering
Predicting the Location and Number of Faults in Large Software Systems
IEEE Transactions on Software Engineering
Software engineering practice versus evidence-based software engineering research
REBSE '05 Proceedings of the 2005 workshop on Realising evidence-based software engineering
Advanced non-distributed operating systems course
ACM SIGCSE Bulletin
IEEE Transactions on Software Engineering
Measuring the Complexity of a UML Component Specification
QSIC '05 Proceedings of the Fifth International Conference on Quality Software
Empirical Software Engineering
Predicting defect densities in source code files with decision tree learners
Proceedings of the 2006 international workshop on Mining software repositories
Simulated annealing for improving software quality prediction
Proceedings of the 8th annual conference on Genetic and evolutionary computation
Looking for bugs in all the right places
Proceedings of the 2006 international symposium on Software testing and analysis
Predicting component failures at design time
Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering
Have things changed now?: an empirical study of bug characteristics in modern open source software
Proceedings of the 1st workshop on Architectural and system support for improving software dependability
Software—Practice & Experience
Reducing TCB complexity for security-sensitive applications: three case studies
Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006
Modeling software testing costs and risks using fuzzy logic paradigm
Journal of Systems and Software
Journal of Systems and Software
Journal of Systems and Software
Data Mining Static Code Attributes to Learn Defect Predictors
IEEE Transactions on Software Engineering
A Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems
IEEE Transactions on Software Engineering
Using Developer Information as a Factor for Fault Prediction
PROMISE '07 Proceedings of the Third International Workshop on Predictor Models in Software Engineering
Modeling the Effect of Size on Defect Proneness for Open-Source Software
PROMISE '07 Proceedings of the Third International Workshop on Predictor Models in Software Engineering
Automating algorithms for the identification of fault-prone files
Proceedings of the 2007 international symposium on Software testing and analysis
Software engineering research: from cradle to grave
Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering
Research issues in software fault categorization
ACM SIGSOFT Software Engineering Notes
Detection and prediction of errors in EPCs of the SAP reference model
Data & Knowledge Engineering
A grid-free abstraction of the Navier-Stokes equations in Fortran 95/2003
ACM Transactions on Mathematical Software (TOMS)
Improving fault detection in modified code: a study from the telecommunication industry
Journal of Computer Science and Technology
A complexity measure for UML component-based system specification
Software—Practice & Experience
The application of product measures in directing software maintenance activity
Journal of Software Maintenance and Evolution: Research and Practice
Mining software repositories for comprehensible software fault prediction models
Journal of Systems and Software
Predicting defect-prone software modules using support vector machines
Journal of Systems and Software
An empirical investigation of software reuse benefits in a large telecom product
ACM Transactions on Software Engineering and Methodology (TOSEM)
Proceedings of the 30th international conference on Software engineering
Predicting defects using network analysis on dependency graphs
Proceedings of the 30th international conference on Software engineering
An initial study of the growth of eclipse defects
Proceedings of the 2008 international working conference on Mining software repositories
Comparing negative binomial and recursive partitioning models for fault prediction
Proceedings of the 4th international workshop on Predictor models in software engineering
A case study in database reliability: component types, usage profiles, and testing
Proceedings of the 1st international workshop on Testing database systems
ACM Transactions on Software Engineering and Methodology (TOSEM)
Improvement of causal analysis using multivariate statistical process control
Software Quality Control
Quantitative analysis of faults and failures with multiple releases of softpm
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement
Exposure model for prediction of number of customer reported defects
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement
Theory of relative defect proneness
Empirical Software Engineering
Empirical Software Engineering
Predicting Defects in Software Using Grammar-Guided Genetic Programming
SETN '08 Proceedings of the 5th Hellenic conference on Artificial Intelligence: Theories, Models and Applications
Can developer-module networks predict failures?
Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering
A defect prediction method for software versioning
Software Quality Control
Towards analysis-driven scientific software architecture: The case for abstract data type calculus
Scientific Programming - Complexity in Scalable Computing
On modeling software defect repair time
Empirical Software Engineering
A case study comparing defect profiles of a reused framework and of applications reusing it
Empirical Software Engineering
Software execution processes as an evolving complex network
Information Sciences: an International Journal
Data mining source code for locating software bugs: A case study in telecommunication industry
Expert Systems with Applications: An International Journal
A tree-based approach to preserve the privacy of software engineering data and predictive models
PROMISE '09 Proceedings of the 5th International Conference on Predictor Models in Software Engineering
Fault detection and prediction in an open-source software project
PROMISE '09 Proceedings of the 5th International Conference on Predictor Models in Software Engineering
A Case Study of Defect Introduction Mechanisms
CAiSE '09 Proceedings of the 21st International Conference on Advanced Information Systems Engineering
Cross-project defect prediction: a large scale experiment on data vs. domain vs. process
Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering
Linux bugs: Life cycle, resolution and architectural analysis
Information and Software Technology
Increasing diversity: Natural language measures for software fault prediction
Journal of Systems and Software
Software testing research and practice
ASM'03 Proceedings of the abstract state machines 10th international conference on Advances in theory and practice
MoDELS'06 Proceedings of the 2006 international conference on Models in software engineering
TestCom'03 Proceedings of the 15th IFIP international conference on Testing of communicating systems
Comparing the effectiveness of several modeling methods for fault prediction
Empirical Software Engineering
What can fault prediction do for you?
TAP'08 Proceedings of the 2nd international conference on Tests and proofs
A complexity reliability model
ISSRE'09 Proceedings of the 20th IEEE international conference on software reliability engineering
Dependability metrics
Testing the theory of relative defect proneness for closed-source software
Empirical Software Engineering
A novel composite model approach to improve software quality prediction
Information and Software Technology
Better, faster, and cheaper: what is better software?
Proceedings of the 6th International Conference on Predictive Models in Software Engineering
On the value of learning from defect dense components for software defect prediction
Proceedings of the 6th International Conference on Predictive Models in Software Engineering
An analysis of developer metrics for fault prediction
Proceedings of the 6th International Conference on Predictive Models in Software Engineering
A modified Yule process to model the evolution of some object-oriented system properties
Information Sciences: an International Journal
Fault tree synthesis from UML models for reliability analysis at early design stages
ACM SIGSOFT Software Engineering Notes
Assessing the current state of software evolution and intellectual energy spent
Proceedings of the Workshop on Advances in Functional Size Measurement and Effort Estimation
Mining software defect data to support software testing management
Applied Intelligence
Information and Software Technology
Program slicing-based cohesion measurement: the challenges of replicating studies using metrics
Proceedings of the 2nd International Workshop on Emerging Trends in Software Metrics
Empirical evaluation of reliability improvement in an evolving software product line
Proceedings of the 8th Working Conference on Mining Software Repositories
Optimizing cost and quality by integrating inspection and test processes
Proceedings of the 2011 International Conference on Software and Systems Process
Defect prediction using social network analysis on issue repositories
Proceedings of the 2011 International Conference on Software and Systems Process
Advances in Engineering Software
An empirical validation of a web fault taxonomy and its usage for web testing
Journal of Web Engineering
Computer Networks: The International Journal of Computer and Telecommunications Networking
Are change metrics good predictors for an evolving software product line?
Proceedings of the 7th International Conference on Predictive Models in Software Engineering
Defect detection effectiveness and product quality in global software development
PROFES'11 Proceedings of the 12th international conference on Product-focused software process improvement
An explanatory analysis on eclipse beta-release bugs through in-process metrics
Proceedings of the 8th international workshop on Software quality
Characteristics of multiple-component defects and architectural hotspots: a large system case study
Empirical Software Engineering
The effectiveness of test-driven development: an industrial case study
Software Quality Control
An investigation on the feasibility of cross-project defect prediction
Automated Software Engineering
Design opportunity tree for schedule management and evaluation by COQUALMO
ICCSA'06 Proceedings of the 2006 international conference on Computational Science and Its Applications - Volume Part IV
Software defect analysis of a multi-release telecommunications system
PROFES'05 Proceedings of the 6th international conference on Product Focused Software Process Improvement
Fault links: exploring the relationship between module and fault types
EDCC'05 Proceedings of the 5th European conference on Dependable Computing
Comparing software measures with fault counts derived from unit-testing of safety-critical software
SAFECOMP'05 Proceedings of the 24th international conference on Computer Safety, Reliability, and Security
Identification of defect-prone classes in telecommunication software systems using design metrics
Information Sciences: an International Journal
Regularities in learning defect predictors
PROFES'10 Proceedings of the 11th international conference on Product-Focused Software Process Improvement
On the use of calling structure information to improve fault prediction
Empirical Software Engineering
Evaluating defect prediction approaches: a benchmark and an extensive comparison
Empirical Software Engineering
Proceedings of the 34th International Conference on Software Engineering
Studying volatility predictors in open source software
Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement
Predicting defect numbers based on defect state transition models
Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement
Measuring architecture quality by structure plus history analysis
Proceedings of the 2013 International Conference on Software Engineering
Predicting bug-fixing time: an empirical study of commercial software projects
Proceedings of the 2013 International Conference on Software Engineering
Using fault history to improve mutation reduction
Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering
Ant colony based approach to predict stock market movement from mood collected on Twitter
Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining
Empirical evidence on OCL formal specification-based metrics as a predictor of fault-proneness
ACM SIGSOFT Software Engineering Notes
ECSA'13 Proceedings of the 7th European conference on Software Architecture
Comparative study on effectiveness of standard bug prediction approaches
Proceedings of the 5th IBM Collaborative Academia Research Exchange Workshop
A study of cyclic dependencies on defect profile of software components
Journal of Systems and Software
An in-depth study of the potentially confounding effect of class size in fault prediction
ACM Transactions on Software Engineering and Methodology (TOSEM)
Empirical evaluation of the understandability of architectural component diagrams
Proceedings of the WICSA 2014 Companion Volume
Bug prediction using entropy-based measures
International Journal of Knowledge Engineering and Data Mining
Applications of fuzzy integrals for predicting software fault-prone
Journal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology
Hi-index | 0.01 |
The dearth of published empirical data on major industrial systems has been one of the reasons that software engineering has failed to establish a proper scientific basis. In this paper, we hope to provide a small contribution to the body of empirical knowledge. We describe a number of results from a quantitative study of faults and failures in two releases of a major commercial system. We tested a range of basic software engineering hypotheses relating to: The Pareto principle of distribution of faults and failures; the use of early fault data to predict later fault and failure data; metrics for fault prediction; and benchmarking fault data. For example, we found strong evidence that a small number of modules contain most of the faults discovered in prerelease testing and that a very small number of modules contain most of the faults discovered in operation. However, in neither case is this explained by the size or complexity of the modules. We found no evidence to support previous claims relating module size to fault density nor did we find evidence that popular complexity metrics are good predictors of either fault-prone or failure-prone modules. We confirmed that the number of faults discovered in prerelease testing is an order of magnitude greater than the number discovered in 12 months of operational use. We also discovered fairly stable numbers of faults discovered at corresponding testing phases. Our most surprising and important result was strong evidence of a counter-intuitive relationship between pre- and postrelease faults: Those modules which are the most fault-prone prerelease are among the least fault-prone postrelease, while conversely, the modules which are most fault-prone postrelease are among the least fault-prone prerelease. This observation has serious ramifications for the commonly used fault density measure. Not only is it misleading to use it as a surrogate quality measure, but, its previous extensive use in metrics studies is shown to be flawed. Our results provide data-points in building up an empirical picture of the software development process. However, even the strong results we have observed are not generally valid as software engineering laws because they fail to take account of basic explanatory data, notably testing effort and operational usage. After all, a module which has not been tested or used will reveal no faults, irrespective of its size, complexity, or any other factor.