A Taxonomy for Classifying Runtime Verification Tools

Abstract

Over the last 15 years Runtime Verification (RV) has grown into a diverse and active field, which has stimulated the development of numerous theoretical frameworks and tools. Many of the tools are at first sight very different and challenging to compare. Yet, there are similarities. In this work, we classify RV tools within a high-level taxonomy of concepts. We first present this taxonomy and discuss the different dimensions. Then, we survey RV tools and classify them according to the taxonomy. This paper constitutes a snapshot of the current state of the art and enables a comparison of existing tools. 1 Introduction Runtime Verification (RV) [7, 28, 29, 38] (or runtime monitoring) is (broadly) the study of methods to analyze the dynamic behavior of computational systems. The most typical analysis is to check whether a given run of a system satisfies a given specification and it is this general setting (and its variants) that we consider in this paper. Whilst topics such as specification mining or trace visualization are generally considered to be within this broad field, we do not include them in our discussion. This paper presents a taxonomy of RV frameworks and tools and uses this to classify 20 selected tools. This work is timely for a number of reasons. Firstly, after more than 15 years of maturing, the field has reached a point where such a general view is needed. The last significant attempt at a taxonomy was in 2004 [24] and had a distinctly different focus to our own. Secondly, a number of activities, such as the runtime verification competitions [4, 6, 30, 48], the RV-CuBES workshop [46, 49], two schools dedicated to RV [16], and a COST action [1] (including the development of a tutorial book on the topic [5]), have put the development of runtime verification tools into focus. Terminology. The field of RV is broad and the used terminology is not yet unified. For the sake of clarity, let us fix the following terms:-Monitored system. The system consisting of software, hardware, or a combination of the two, that is being monitored. Its behavior is usually abstracted as a trace object. The authors warmly thank Martin Leucker for the early discussions on the taxonomy and mind map representation. This article is based upon work from COST Action ARVI IC1402, supported by COST (European Cooperation in Science and Technology). In particular, the taxonomy and classification benefited from discussions within working groups one and two of this action. We would also like to acknowledge input from participants of Dagstuhl seminar 17462 [34].

RML: runtime monitoring language

“…This is not meant to be a precise, complete classification of runtime verification tools. This is a complex research task by itself; such a taxonomy has been recently proposed by Falcone, Krstic, et al (2018), and a graphical representation is given in Figure 7.1. With respect to such taxonomy, we can identify RML as follows.…”

“…However, such outcomes are delivered to the instrumentation layer, which may be either passive or active; the latter would affect the execution of the program under scrutiny to some extent. (Falcone, Krstic, et al, 2018).…”

“…"A formalism for specification of Java API interfaces." In: Companion Proceedings for the ISSTA/ECOOP 2018Workshops, ISSTA 2018, Amsterdam, Netherlands, July 16-21, 2018 (Falcone, Krstic, et al, 2018). 72 L I S T I N G S…”

COST Action IC1402 Runtime Verification Beyond Monitoring

“…-A taxonomy of RV aspects that "paves the road" to allow a classification and comparison of theoretical results, problems and techniques. The taxonomy has been published in [16]. -The identification of the challenges and opportunities of instrumentation, where the system under scrutiny is modified or harnessed to allow the monitoring process.…”

“…-Classification of Tools. The taxonomy mentioned above (in working group 1) was developed alongside a classification of Runtime Verification tools and further refined with respect to this classification [16]. -Exploration of Language Landscape.…”