What We Know about Testing Embedded Software

Abstract

Abstract. Embedded systems have overwhelming penetration around the world. Innovations are increasingly triggered by software embedded in automotive, transportation, medical-equipment, communication, energy, and many other types of systems. To test embedded software in a cost effective manner, a large number of test techniques, approaches, tools and frameworks have been proposed by both practitioners and researchers in the last several decades. However, reviewing and getting an overview of the entire state-of-theart and the -practice in this area is challenging for a practitioner or a (new) researcher. Also unfortunately, we often see that some companies reinvent the wheel (by designing a test approach new to them, but existing in the domain) due to not having an adequate overview of what already exists in this area. To address the above need, we conducted a systematic literature review (SLR) in the form of a systematic mapping (classification) in this area. After compiling an initial pool of 560 papers, a systematic voting was conducted among the authors, and our final pool included 272 technical papers. The review covers the types of testing topics studied, types of testing activity, types of test artifacts generated (e.g., test inputs or test code), and the types of industries in which studies have focused on, e.g., automotive and home appliances. Our article aims to benefit the readers (both practitioners and researchers) by serving as an "index" to the vast body of knowledge in this important and fast-growing area.

A survey on software testability

“…To put the annual trend of papers in this area in perspective, we show, also in Figure 14, a similar chart reported by another recent survey study in the area of testing embedded software [12]. It is interesting to observe that while for the area of testing embedded software, the number of papers is generally increasing, for papers in the area of software testability, there are ups and downs, perhaps denoting a continuous change in challenges and/or interest in this area over the years.…”

“…As a point of reference and for comparison, we show in Figure 8 the same chart from our other recent SLM study on testing embedded software [12]. By comparing the two charts ( Figure 7 and Figure 8), we can see that the area of software testability, as a whole, puts more importance on conducting empirical studies than the area of testing embedded software.…”

“…To put things in perspective, we also compare the trend with data from five other SLM/SLR studies: (1) a SLM on web application testing [54], (2) a SLM on testing embedded software [12], (3) a SLM on Graphical User Interface (GUI) testing [55], (4) a survey on search-based testing (SBST) [56], and (5) a survey on mutation testing [57]. Note that the data for the other areas are not until year 2017, since the execution and publication timelines of those survey papers are in earlier years, e.g., the survey on mutation testing [57] was published in 2011 and thus only has the data until 2009.…”

“…Approaches to quantitative and qualitative measurement of testability are very prominent and address all phases of development. (21), adding assertions (16), and improving controllability (12). Furthermore, specific architecture and/or test interfaces supporting testability (6) and manipulating dependency (6) are proposed.…”

“…Our review pool included 208 technical papers published in conferences and journals, and the earliest paper [P109] 1 was published in 1982, entitled "On testing non-testable programs" which highlights the significance of testability more than three decades ago. Previous "review" (survey) papers like this article have appeared on other topics, e.g., Agile development [10], developer motivation [11], or testing embedded software [12], and have shown to be useful in providing concise overviews on a given area.…”