Model-based testing of NASA’s GMSEC, a reusable framework for ground system software

Abstract

We present an empirical study in which model-based testing (MBT) was applied to the software bus of NASA’s Goddard Mission Service Evolution Center (GMSEC), a reusable software framework. The goal was to study the feasibility of using MBT on a real-world software system that was designed to be flexible. GMSEC has three levels of flexibility: 1) loose application coupling through a software bus based on the publish–subscribe architectural style, 2) language independence by providing APIs to the bus in several programming languages, 3) middleware independence by providing wrappers for several middlewares that are supported by the software bus. The novelty brought forward in this paper is that one model and one set of generated test cases were used as the basis to test the software bus for behavioral consistency across multiple programming languages and middleware wrappers. The comparison of costs and benefits from using finite state machines (FSM) vs. extended FSMs (EFSM) when used for MBT on a real-world system is also novel. The case study shows that it was feasible, even for a programmer who neither knew MBT nor the system under test, to successfully apply MBT to a flexible system such as GMSEC and that MBT could within reasonable effort detect non-trivial defects in a fielded system

Model-based Testing of Mobile Systems – An Empirical Study on QuizUp Android App

“…See Table 2 for effort spent before modeling and testing the QuizUp application. We had learned MBT and the Graphwalker tool through the GMSEC study [10], therefore, no such effort was required for this study. However, applying MBT on a mobile application was new to us and we had to familiarize ourselves with the Appium test driver.…”

“…We will now compare the QuizUp results to our most recent study on GMSEC [10]. At the time of the QuizUp study, we had gained experience in MBT which meant that we were much more efficient in setting up the QuizUp study (3 weeks compared to 7 weeks in the previous study).…”

“…We learned from the GMSEC case study that non-executable mapping tables can be messy and that maintaining a single mapping table becomes challenging when the model size increases [10]. Thus, we decided to implement a class structure which would allow us to have a separate mapping table for each scene.…”

“…In previous work [9,10,13], we have used model-based testing (MBT) to test non-mobile systems because MBT addresses the problem of manually creating test cases. Instead of creating one test case at a time, MBT builds a model of the system under test, which is then used to automatically generate executable test cases.…”