Imaging the earth's interior
The seismic kernel system—a large-scale exercise in Fortran 77 portability
Software—Practice & Experience
Risks to the public in computers and related systems
ACM SIGSOFT Software Engineering Notes
Safer C: Developing Software for in High-Integrity and Safety-Critical Systems
Safer C: Developing Software for in High-Integrity and Safety-Critical Systems
The philosophy of nothing and everything
APL '95 Proceedings of the international conference on Applied programming languages
An empirical study of static call graph extractors
Proceedings of the 18th international conference on Software engineering
An empirical study of static call graph extractors
ACM Transactions on Software Engineering and Methodology (TOSEM)
The T Experiments: Errors In Scientific Software
IEEE Computational Science & Engineering
How Goes CSE? Thoughts on the IEEE CS Workshop at Purdue
IEEE Computational Science & Engineering
N-Version Design Versus One Good Version
IEEE Software
The Reliability of Diverse Systems: A Contribution Using Modelling of the Fault Creation Process
DSN '01 Proceedings of the 2001 International Conference on Dependable Systems and Networks (formerly: FTCS)
Sound methods and effective tools for engineering modeling and analysis
Proceedings of the 25th International Conference on Software Engineering
Value-maximizing deadline scheduling and its application to animation rendering
Proceedings of the seventeenth annual ACM symposium on Parallelism in algorithms and architectures
Testing for trustworthiness in scientific software
SECSE '09 Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering
A document driven methodology for developing a high quality Parallel Mesh Generation Toolbox
Advances in Engineering Software
A learning-based approach to unit testing of numerical software
ICTSS'10 Proceedings of the 22nd IFIP WG 6.1 international conference on Testing software and systems
Scientific software production: incentives and collaboration
Proceedings of the ACM 2011 conference on Computer supported cooperative work
Trustworthy numerical computation in Scala
Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications
An Analysis of Process Characteristics for Developing Scientific Software
Journal of Organizational and End User Computing
| Hi-index | 0.01 |
This paper describes some results of what, to the authors' knowledge, is the largest N-version programming experiment ever performed. The object of this ongoing four-year study is to attempt to determine just how consistent the results of scientific computation really are, and, from this, to estimate accuracy. The experiment is being carried out in a branch of the earth sciences known as seismic data processing, where 15 or so independently developed large commercial packages that implement mathematical algorithms from the same or similar published specifications in the same programming language (Fortran) have been developed over the last 20 years. The results of processing the same input dataset, using the same user-specified parameters, for nine of these packages is reported in this paper. Finally, feedback of obvious flaws was attempted to reduce the overall disagreement. The results are deeply disturbing. Whereas scientists like to think that their code is accurate to the precision of the arithmetic used, in this study, numerical disagreement grows at around the rate of 1% in average absolute difference per 4000 fines of implemented code, and, even worse, the nature of the disagreement is nonrandom. Furthermore, the seismic data processing industry has better than average quality standards for its software development with both identifiable quality assurance functions and substantial test datasets.