Characteristics, potentials, and limitations of open-source Simulink projects for empirical research

Abstract

Simulink is an example of a successful application of the paradigm of model-based development into industrial practice. Numerous companies create and maintain Simulink projects for modeling software-intensive embedded systems, aiming at early validation and automated code generation. However, Simulink projects are not as easily available as code-based ones, which profit from large publicly accessible open-source repositories, thus curbing empirical research. In this paper, we investigate a set of 1734 freely available Simulink models from 194 projects and analyze their suitability for empirical research. We analyze the projects considering (1) their development context, (2) their complexity in terms of size and organization within projects, and (3) their evolution over time. Our results show that there are both limitations and potentials for empirical research. On the one hand, some application domains dominate the development context, and there is a large number of models that can be considered toy examples of limited practical relevance. These often stem from an academic context, consist of only a few Simulink blocks, and are no longer (or have never been) under active development or maintenance. On the other hand, we found that a subset of the analyzed models is of considerable size and complexity. There are models comprising several thousands of blocks, some of them highly modularized by hierarchically organized Simulink subsystems. Likewise, some of the models expose an active maintenance span of several years, which indicates that they are used as primary development artifacts throughout a project’s lifecycle. According to a discussion of our results with a domain expert, many models can be considered mature enough for quality analysis purposes, and they expose characteristics that can be considered representative for industry-scale models. Thus, we are confident that a subset of the models is suitable for empirical research. More generally, using a publicly available model corpus or a dedicated subset enables researchers to replicate findings, publish subsequent studies, and use them for validation purposes. We publish our dataset for the sake of replicating our results and fostering future empirical research.

SLNET: A Redistributable Corpus of 3rd-party Simulink Models

“…SLNET is not a superset of earlier Simulink corpora [4,11] as earlier corpora were neither self-contained nor redistributable. Figure 2 gives an overview of SLNET's construction.…”

“…Users can thus select models and projects from SLNET via SQL queries. To get an insight into the projects' domain and popularity we first searched the user-generated project tags (i.e., GitHub "topics" and MATLAB Central "categories") for common domains (i.e., the Simulink project domains identified by Boll et al [4]), yielding Electronics (983), Automotive (64), Communications (61), Robotics (52), Energy (48), Aerospace (47), Biotech (20), and Medicine (2). Table 2 shows data often used as proxies for project popularity or engagement (e.g., people who have star-ed or forked a GitHub project or provided a 1-5 star rating for a MATLAB Central project).…”

“…The most closely related previous work has studied an initial collection of 391 third-party Simulink models [9] and later extended it to a curated corpus ("SLC0") of some 1k third-party Simulink models [11]. Boll et al [4] collected an updated version of SLC0 and assessed the corpus's suitability for empirical research. While pioneering larger studies and validating that models from such a corpus can be similar to industrial models, these collections consisted of a list of URLs to non-permanent resources [9] and contained models with unclear license information [11].…”

“…To allow fine-grained selection of Simulink models and projects, SLNET computes several project-level and model-level metrics [4] and exposes them in a SQL database. SLNET similarly identifies and labels libraries and models that are test harnesses [31].…”

Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions

“…In addition, we define ATheNA-S, an instance of ATheNA that supports Simulink ® models. We consider Simulink ® models since they are widely-used for specifying the behavior of CPSs in a variety of domains including [21,55], automotive [62], energy [47] and medical [82]. We implement ATheNA-S as a plugin for S-Taliro [10], a well-known SBST framework for Simulink ® models recently classified as ready for industrial usage [48].…”