Evaluating applicability of combinatorial testing in an industrial environment: a case study

Abstract

Case studies for evaluating tools in software engineering are powerful. Although they cannot achieve the scientific rigour of formal experiments, the results can provide sufficient information to help companies judge if a specific technology being evaluated will benefit their organization. This paper reports on a case study done for evaluating a combinatorial testing tool in a realistic industrial environment with real objects and subjects. The case study has been executed at Sulake, a company that develops social entertainment games and whose main product is Habbo Hotel, a social network community in the shape of an online Hotel that is visited by millions of teenagers every week all around the world. This paper describes the experimental design of the case study together with the results and decisions that Sulake has taken about the further study, adoption and implantation of these type of tools.

A Study of Performance Testing in Configurable Software Systems

“…Configuration Space The configuration space has been studied in prior work [12] [22] [23]. Nair et al [22] discuss the residual-based and rank-based approaches to sample the configuration space.…”

“…Nair et al [22] discuss the residual-based and rank-based approaches to sample the configuration space. Puoskari et al [23] conduct a combinatorial testing case study in an industrial environment. Reisner et al [12] conduct an empirical study of configuration space explosion in configurable software systems.…”

Evaluation of the IPO-Family algorithms for test case generation in web security testing

“…Moreover, a proof-of-concept experiment using a partial t-wise coverage framework to analyze integration and test data from three different NASA spacecrafts has been presented in [17]. Finally, combinatorial testing on ID3v2 tags of MP3 files was given in [18] while a case study for evaluating the applicability of combinatorial testing in an industrial environment was presented in [19].…”

Automating Test Case Design within the Classification Tree Editor

“…A body of work on the application of the classification tree method and CTE can be found in recent work [34]- [37].…”

“…CONCLUSION Regardless of the system under test, level of integration, test phase and domain, the classification tree editor is universally applicable [34]- [37]. This method has found a worldwide acceptance over the last two decades and has been used by commercial data processing, aviation and aerospace industries, and many others, for example, for examining the Hubble Space Telescope [45].…”