Analysis of faults in an N-version software experiment

Abstract

We have conducted a large-scale experiment in N-version programming. A total of 27 versions of a program were prepared independently from the same specification at two universities. The results of executing the versions revealed that the versions were individually extremely reliable but that the number of input cases in which more than one failed was substantially more than would be expected if they were statistically independent. After the versions had been executed, the failures of each version were examined and the associated faults located. In this paper we present an analysis of these faults. Our goal in undertaking this analysis was to understand better the nature of the faults. We found that in some cases the programmers made equivalent logical errors, indicating that some parts of the problem were simply more difficult than others. We also found cases in which apparently different logical errors yielded faults that caused statistically correlated failures, indicating that there are special cases in the input space that present difficulty in various parts of the solution. A formal model is presented to explain this phenomenon. It appears that minor differences in the software development environment, such as the use of different programming languages for the different versions, would not have a major impact in reducing the incidence of faults that cause correlated failures.

DAFT: Decoupled Acyclic Fault Tolerance

“…Lessons from n-version programming can also be applied to fault detection [1,2,10,15]. Process-level Redundancy (PLR) presented by Shye et al acts as a shim between user programs and the operating system [15].…”

The System-Level Simplex Architecture for Improved Real-Time Embedded System Safety

“…Then, all are run and the result given by the majority of versions is taken as the output of the system. One drawback with this method is the lack of statistical independence of bugs [23], [24]. Additionally, for a constant amount of development effort, N-version programming is actually less reliable than focusing on a single version over a wide range of parameter values [1].…”

Model implementation of the simulation environment of voting algorithms, as a dynamic system for increasing the reliability of the control complex of autonomous unmanned objects

“…We take into account the situation when the algorithm version worked correctly, but it gave an inaccurate response due to rounding faults, digitization, etc. [11] The fault appears with the probability given for each version and each data stream. In the event of a fault, the following checks occur: if it's not first fault in current voting, in this case an related fault will be generated with predetermined probability, i.e.…”