Scientific Software Testing Goes Serverless: Creating and Invoking Metamorphic Functions

Abstract

Our function-as-a-service (FaaS) framework transformed end users' questions into automated tests for scientific software. Our case study illustrates the FaaSification of scientific software testing and the importance of value-driven evaluations by focusing on real-world defect detection.MANY SCIENTIFIC RESEARCHERS rely on software to perform their research. In climate research, for instance, scientists and policy makers rely on code simulations to inform their recommendations and decisions. 1 Because scientific software needs to produce trustworthy results, rigorous software engineering practices should be adopted to assure quality.Although testing is a mainstream approach toward software quality, scientific software developers face many challenges, 2 one of which is that only a small group of scientific domain experts (SDEs) who are software developers themselves could write the tests in relation to the scientific theories and models, and yet their primary goal is to further science. 3 Fortunately, science is a community endeavor, and the user community is the cornerstone underlying the sustainable development of any scientific software. However, without the proper tools, serving the user community poses additional burdens on the SDE developers.To alleviate the burdens, we present a function-as-a-service (FaaS) framework aiming to support the SDEs in efficiently processing software users' questions and further transforming the questions into automatic tests. Because users raise their concerns on an asneeded basis, building a monolithic service (for example, via the traditional web clientserver architecture) with availability and capacity would not be cost-effective. FaaS breaks the monolith into functions that can be separately instantiated, executed, and destroyed, and hence, reduces the operational cost. The functions in our work are extracted from the software forum's posts and transformed into system-level test cases. For example, the two functions, increase_bio_retention_cells() and decrease_ runoff(), are generated from the post expressing, "the simulated surface water runoff is expected to decrease when bioretention cells are added to an area of study."

Testing Ocean Software with Metamorphic Testing

“…For instances, in testing simulations, MT has been successfully adopted to test agent-based (ABM) and discrete-event (DES) simulations [26], hybrid ABM and DES systems [11], health care simulation [25], webenabled simulation [1], and simulator platform for self-driving cars [36,41]. In testing scientific software, MT is an effective technique to detect faults in simulation programs for designing nuclear power plants [14], bioinformatics programs [4], epidemiological models [31,33], chemical reaction networks for prototyping nano-scale molecular devices [13], matrix calculation programs [32], solvers for partial differential equations [3], multiple linear regression software [22], ocean modelling [16], storm water management model systems [19], machine learning-based hydro-logical models [40], Monte-Carlo computational programs [10,30], serverless scientific applications [20], as well as other types of scientific software [9,18,19,29]. For example, He et al [14] has found 33 bugs in simulation programs that are used to design and analyze nuclear power plants in a study that adopts MT.…”

Reliability Assessment of Machine Learning Models in Hydrological Predictions Through Metamorphic Testing

“…Although there are more studies on testing hydrological modeling software using MT, such as Lin, Peng, et al. (2021) and Lin, Simon, and Niu (2021), the MRs used in these studies were not derived directly from hydrological knowledge of the system being modeled.…”