Runtime Verification with State Estimation

Abstract

Abstract. We introduce the concept of Runtime Verification with StateEstimation and show how this concept can be applied to estimate the probability that a temporal property is satisfied by a run of a program when monitoring overhead is reduced by sampling. In such situations, there may be gaps in the observed program executions, thus making accurate estimation challenging. To deal with the effects of sampling on runtime verification, we view event sequences as observation sequences of a Hidden Markov Model (HMM), use an HMM model of the monitored program to "fill in" sampling-induced gaps in observation sequences, and extend the classic forward algorithm for HMM state estimation (which determines the probability of a state sequence, given an observation sequence) to compute the probability that the property is satisfied by an execution of the program. To validate our approach, we present a case study based on the mission software for a Mars rover. The results of our case study demonstrate high prediction accuracy for the probabilities computed by our algorithm. They also show that our technique is much more accurate than simply evaluating the temporal property on the given observation sequences, ignoring the gaps.

Distributed Finite-State Runtime Monitoring with Aggregated Events

“…Statistical methods can then be employed to infer the most probable sequence of state transitions that occurred during the time in which sampling was suspended. For instance, Stoller et al [12] consider the scenario where instrumentation is suspended for some period of time, leaving a gap in the sequence of observed events.…”

“…Discarding ordering information allows event streams to be compressed effectively, whilst retaining event frequencies and types maintains a certain level of precision. In comparison to related work [5,12], this trace model is not probabilistic and does not allow for "gaps" in the event stream-every occurring event is indeed accounted for. The aggregated trace rather provides an over-approximation that implicitly includes all permutations of the original trace it represents.…”

Runtime Verification with Particle Filtering

“…We developed a framework for this, called Runtime Verification with State Estimation (RVSE) [10], in which a hidden Markov model (HMM) is used to succinctly model the program and compute the uncertainty in predictions due to incomplete information.…”

“…In contrast to our previous work [10,1], we model the HMM, the DFA, and their composition in a more elegant and succinct way as a dynamic Bayesian network (DBN). This allows us to properly formalize a new kind of event, called peek events, which are inexpensive observations of part of the program state.…”

“…Runtime Verification with State Estimation (RVSE) [10] is an algorithm for runtime verification in the presence of observation gaps. In RVSE, a Hidden Markov Model of the monitored program is constructed, monitored event sequences are treated as observation sequences of the HMM, and an extension of the optimal forward algorithm for HMM state estimation [7] is used to estimate the state of the HMM and the monitor DFA.…”