Software safety: why, what, and how
ACM Computing Surveys (CSUR)
The consistent comparison problem in N-version software
ACM SIGSOFT Software Engineering Notes
Social choice theory and distributed decision making
COCS '88 Proceedings of the ACM SIGOIS and IEEECS TC-OA 1988 conference on Office information systems
Data Diversity: An Approach to Software Fault Tolerance
IEEE Transactions on Computers - Fault-Tolerant Computing
Conceptual Modeling of Coincident Failures in Multiversion Software
IEEE Transactions on Software Engineering
The Consistent Comparison Problem in N-Version Software
IEEE Transactions on Software Engineering
The Use of Self Checks and Voting in Software Error Detection: An Empirical Study
IEEE Transactions on Software Engineering
Modeling of Correlated Failures and Community Error Recovery in Multiversion Software
IEEE Transactions on Software Engineering
Analysis of Faults in an N-Version Software Experiment
IEEE Transactions on Software Engineering
Resourceful systems for fault tolerance, reliability, and safety
ACM Computing Surveys (CSUR)
Evaluation of safety-critical software
Communications of the ACM
Software safety in embedded computer systems
Communications of the ACM
An Environment for Developing Fault-Tolerant Software
IEEE Transactions on Software Engineering
An Empirical Comparison of Software Fault Tolerance and Fault Elimination
IEEE Transactions on Software Engineering
An Experimental Evaluation of Software Redundancy as a Strategy for Improving Reliability
IEEE Transactions on Software Engineering
An experimental comparison of the effectiveness of the all-uses and all-edges adequacy criteria
TAV4 Proceedings of the symposium on Testing, analysis, and verification
The infeasibility of experimental quantification of life-critical software reliability
SIGSOFT '91 Proceedings of the conference on Software for citical systems
Performance Analysis of Real-Time Software Supporting Fault-Tolerant Operation
IEEE Transactions on Computers
A New Approach to the Modeling of Recovery Block Structures
IEEE Transactions on Software Engineering
High-pressure steam engines and computer software
ICSE '92 Proceedings of the 14th international conference on Software engineering
Estimation and Enhancement of Real-Time Software Reliability Through Mutation Analysis
IEEE Transactions on Computers - Special issue on fault-tolerant computing
Modeling Correlation in Software Recovery Blocks
IEEE Transactions on Software Engineering - Special issue on software reliability
A reliability model combining representative and directed testing
Proceedings of the 18th international conference on Software engineering
Reliable software and communication: software quality, reliability, and safety
ICSE '93 Proceedings of the 15th international conference on Software Engineering
Highly reliable upgrading of components
Proceedings of the 21st international conference on Software engineering
Toward a resourceful method of software fault tolerance
ACM-SE 37 Proceedings of the 37th annual Southeast regional conference (CD-ROM)
Automatically Checking an Implementation against Its Formal Specification
IEEE Transactions on Software Engineering
Empirical studies of software engineering: a roadmap
Proceedings of the Conference on The Future of Software Engineering
Communications of the ACM
Eliminating Exception Handling Errors with Dependability Cases: A Comparative, Empirical Study
IEEE Transactions on Software Engineering
Modeling the Effects of Combining Diverse Software Fault Detection Techniques
IEEE Transactions on Software Engineering
The Use of Proof in Diversity Arguments
IEEE Transactions on Software Engineering - special section on current trends in exception handling—part II
Increasing the confidence in off-the-shelf components: a software connector-based approach
SSR '01 Proceedings of the 2001 symposium on Software reusability: putting software reuse in context
Theory of software reliability based on components
ICSE '01 Proceedings of the 23rd International Conference on Software Engineering
Modeling software design diversity: a review
ACM Computing Surveys (CSUR)
Heterogeneous networking: a new survivability paradigm
Proceedings of the 2001 workshop on New security paradigms
Applying data redundancy to differential equation solvers
Annals of Software Engineering
Modeling reliability growth during non-representative
Annals of Software Engineering
Relational programs: An architecture for robust real-time safety-critical process-control systems
Annals of Software Engineering
Splitting the Difference: The Historical Necessity of Synthesis in Software Engineering
IEEE Annals of the History of Computing
The T Experiments: Errors In Scientific Software
IEEE Computational Science & Engineering
Implementing Design Diversity to Achieve Fault Tolerance
IEEE Software
N-Version Design Versus One Good Version
IEEE Software
Exploring the Role of Diagnosis in Software Engineering
IEEE Software
The Infeasibility of Quantifying the Reliability of Life-Critical Real-Time Software
IEEE Transactions on Software Engineering
An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing
IEEE Transactions on Software Engineering
A Controlled Experiment in Maintenance Comparing Design Patterns to Simpler Solutions
IEEE Transactions on Software Engineering
Design and Verification of Distributed Recovery Blocks with CSP
Formal Methods in System Design
N-Version Genetic Programming via Fault Masking
EuroGP '02 Proceedings of the 5th European Conference on Genetic Programming
A Constant Perturbation Method for Evaluation of Structural Diversity in Multiversion Software
SAFECOMP '00 Proceedings of the 19th International Conference on Computer Safety, Reliability and Security
Assessment of the Reliability of Fault-Tolerant Software: A Bayesian Approach
SAFECOMP '00 Proceedings of the 19th International Conference on Computer Safety, Reliability and Security
The Reliability of Diverse Systems: A Contribution Using Modelling of the Fault Creation Process
DSN '01 Proceedings of the 2001 International Conference on Dependable Systems and Networks (formerly: FTCS)
Estimating Bounds on the Reliability of Diverse Systems
IEEE Transactions on Software Engineering
Improving generalization of neural networks using multilayer perceptron discriminants
Systems Analysis Modelling Simulation - Special issue: Advances in control and computer engineering
Statecharts Supervision Models for Soft Real-Time Systems
ASSET '98 Proceedings of the 1998 IEEE Workshop on Application - Specific Software Engineering and Technology
Diversity in the Software Development Process
WORDS '97 Proceedings of the 3rd Workshop on Object-Oriented Real-Time Dependable Systems - (WORDS '97)
An empirical evaluation of maximum likelihood voting in failure correlation conditions
ISSRE '96 Proceedings of the The Seventh International Symposium on Software Reliability Engineering
Reducing Uncertainty About Common-Mode Failures
ISSRE '97 Proceedings of the Eighth International Symposium on Software Reliability Engineering
An Approach to Measuring and Assessing Dependability for Critical Software Systems
ISSRE '97 Proceedings of the Eighth International Symposium on Software Reliability Engineering
FTCS '95 Proceedings of the Twenty-Fifth International Symposium on Fault-Tolerant Computing
Behavioral Diversity and a Probabilistically Optimal GP Ensemble
Genetic Programming and Evolvable Machines
Optimal testing-resource allocation with genetic algorithm for modular software systems
Journal of Systems and Software
Separating agreement from execution for byzantine fault tolerant services
SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
The Knowledge Engineering Review
Software assurance by bounded exhaustive testing
ISSTA '04 Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis
Towards a Control-Theoretical Approach to Software Fault-Tolerance
QSIC '04 Proceedings of the Quality Software, Fourth International Conference
Review and analysis of synthetic diversity for breaking monocultures
Proceedings of the 2004 ACM workshop on Rapid malcode
The Ranger Robotic Satellite Servicer and Its Autonomous Software-Based Safety System
IEEE Intelligent Systems
Software dissimilarity debate revisited
ACM SIGSOFT Software Engineering Notes
Software Assurance by Bounded Exhaustive Testing
IEEE Transactions on Software Engineering
A Comprehensive Model for Software Rejuvenation
IEEE Transactions on Dependable and Secure Computing
Leveraging architectural models to inject trust into software systems
SESS '05 Proceedings of the 2005 workshop on Software engineering for secure systems—building trustworthy applications
Implementing Trustworthy Services Using Replicated State Machines
IEEE Security and Privacy
A relational database model of program execution and software components
ACM-SE 38 Proceedings of the 38th annual on Southeast regional conference
Diversity as a computer defense mechanism
NSPW '05 Proceedings of the 2005 workshop on New security paradigms
Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications
Comparison-based file server verification
ATEC '05 Proceedings of the annual conference on USENIX Annual Technical Conference
Safety and Software Intensive Systems: Challenges Old and New
FOSE '07 2007 Future of Software Engineering
IEEE Transactions on Software Engineering
Do programmer pairs make different mistakes than solo programmers?
Journal of Systems and Software
Zyzzyva: speculative byzantine fault tolerance
Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles
Fault Tolerance via Diversity for Off-the-Shelf Products: A Study with SQL Database Servers
IEEE Transactions on Dependable and Secure Computing
Information Assurance: Dependability and Security in Networked Systems
Information Assurance: Dependability and Security in Networked Systems
Engineering multiversion neural-net systems
Neural Computation
Setting checkpoints in legacy code to improve fault-tolerance
Journal of Systems and Software
Reliable Software Systems Design: Defect Prevention, Detection, and Containment
Verified Software: Theories, Tools, Experiments
Ontology Supported Selection of Versions for N-Version Programming in Semantic Web Services
ICCS '08 Proceedings of the 8th international conference on Computational Science, Part I
Uncertainty explicit assessment of off-the-shelf software: A Bayesian approach
Information and Software Technology
A simple totally ordered broadcast protocol
LADIS '08 Proceedings of the 2nd Workshop on Large-Scale Distributed Systems and Middleware
Automatic system testing of programs without test oracles
Proceedings of the eighteenth international symposium on Software testing and analysis
Software, software engineering and software engineering research: some unconventional thoughts
Journal of Computer Science and Technology
N-version programming with imperfect debugging
Computers and Electrical Engineering
Computational Epistemology and e-Science: A New Way of Thinking
Minds and Machines
NSPW '09 Proceedings of the 2009 workshop on New security paradigms workshop
Improving dependability of component-based systems via multi-versioning connectors
Architecting dependable systems
Insights on fault interference for programs with multiple bugs
ISSRE'09 Proceedings of the 20th IEEE international conference on software reliability engineering
Dependability metrics
Modelling the effects of combining diverse software fault detection techniques
Formal methods and testing
MiTV: multiple-implementation testing of user-input validators for web applications
Proceedings of the IEEE/ACM international conference on Automated software engineering
Increasing web service dependability through consensus voting
COMPSAC-W'05 Proceedings of the 29th annual international conference on Computer software and applications conference
Structural analysis of explicit fault-tolerant programs
HASE'04 Proceedings of the Eighth IEEE international conference on High assurance systems engineering
E unibus pluram: massive-scale software diversity as a defense mechanism
Proceedings of the 2010 workshop on New security paradigms
Runtime verification in context: can optimizing error detection improve fault diagnosis?
RV'10 Proceedings of the First international conference on Runtime verification
Testing and validating machine learning classifiers by metamorphic testing
Journal of Systems and Software
On the estimation of adequate test set size using fault failure rates
Journal of Systems and Software
Discrepancy discovery in search-enhanced testing
Proceedings of the 3rd International Workshop on Search-Driven Development: Users, Infrastructure, Tools, and Evaluation
Search-enhanced testing (NIER track)
Proceedings of the 33rd International Conference on Software Engineering
On effective testing of health care simulation software
Proceedings of the 3rd Workshop on Software Engineering in Health Care
Finding and understanding bugs in C compilers
Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation
Automating image segmentation verification and validation by learning test oracles
Information and Software Technology
Tradeoff exploration between reliability, power consumption, and execution time
SAFECOMP'11 Proceedings of the 30th international conference on Computer safety, reliability, and security
The use of mathematics in software quality assurance
Frontiers of Computer Science in China
The effectiveness of choice of programming language as a diversity seeking decision
EDCC'05 Proceedings of the 5th European conference on Dependable Computing
Dependable and Historic Computing
Dependable and Historic Computing
Beyond traces and independence
Dependable and Historic Computing
Do programmer pairs make different mistakes than solo programmers?
EASE'06 Proceedings of the 10th international conference on Evaluation and Assessment in Software Engineering
Checking formal specifications by testing
IW-FM'99 Proceedings of the 3rd Irish conference on Formal Methods
Journal of Data and Information Quality (JDIQ)
DEX: Digital evidence provenance supporting reproducibility and comparison
Digital Investigation: The International Journal of Digital Forensics & Incident Response
An experimental comparison of real and artificial deception using a deception generation model
Decision Support Systems
Practical hardening of crash-tolerant systems
USENIX ATC'12 Proceedings of the 2012 USENIX conference on Annual Technical Conference
Achieving highly reliable embedded software: an empirical evaluation of different approaches
SAFECOMP'07 Proceedings of the 26th international conference on Computer Safety, Reliability, and Security
Safety demonstration and software development
SAFECOMP'07 Proceedings of the 26th international conference on Computer Safety, Reliability, and Security
Turtles all the way down: a clean-slate, ground-up, first-principles approach to secure systems
Proceedings of the 2012 workshop on New security paradigms
ChameleonSoft: Software Behavior Encryption for Moving Target Defense
Mobile Networks and Applications
Reliability and performance optimization of pipelined real-time systems
Journal of Parallel and Distributed Computing
Proceedings of the 17th Conference on Pattern Languages of Programs
| Hi-index | 0.06 |
N-version programming has been proposed as a method of incorporating fault tolerance into software. Multiple versions of a program (i.e., “N”) are prepared and executed in parallel. Their outputs are collected and examined by a voter, and, if they are not identical, it is assumed that the majority is correct. This method depends for its reliability improvement on the assumption that programs that have been developed independently will fail independently. In this paper an experiment is described in which the fundamental axiom is tested. A total of 27 versions of a program were prepared independently from the same specification at two universities and then subjected to one million tests. The results of the tests revealed that the programs were individually extremely reliable but that the number of tests in which more than one program failed was substantially more than expected. The results of these tests are presented along with an analysis of some of the faults that were found in the programs. Background information on the programmers used is also summarized. The conclusion from this experiment is that N-version programming must be used with care and that analysis of its reliability must include the effect of dependent errors.