Test confessions: A study of testing practices for plug-in systems

Abstract

Abstract-Testing plug-in-based systems is challenging due to complex interactions among many different plug-ins, and variations in version and configuration. The objective of this paper is to increase our understanding of what testers and developers think and do when it comes to testing plug-inbased systems. To that end, we conduct a qualitative (grounded theory) study, in which we interview 25 senior practitioners about how they test plug-in applications based on the Eclipse plug-in architecture. The outcome is an overview of the testing practices currently used, a set of identified barriers limiting test adoption, and an explanation of how limited testing is compensated by self-hosting of projects and by involving the community. These results are supported by a structured survey of more than 150 professionals. The study reveals that unit testing plays a key role, whereas plug-in specific integration problems are identified and resolved by the community. Based on our findings, we propose a series of recommendations and areas for future research.

Exploring variability-aware execution for testing plugin-based web applications

“…It is well-known for being prone to plugin conflicts [6]. It exhibits common plugin conflict characteristics as studied in other plugin systems [32].…”

“…Beyond opportunistic and ad-hoc testing of individual configurations [32,43] How to write such test cases is well established and common practice [9]. As an illustration, in Figure 4a we show a test case for the Weather plugin querying the main page and checking whether the '[:weather:]' tag has been replaced correctly and whether the jQuery library has an up-to-date version.…”

“…pairwise sampling [43,24], can help to reduce the test effort, but are intrinsically incomplete. Shipping test cases with the plugins [32] delays test execution until a user actually combines plugins, delays the detection and resolution of conflicts, and shifts responsibilities to the users.…”

“…The developers could write test cases and execute them on individual plugin configurations to detect conflicts. However, executing the test cases for plugins is challenging due to the combinatorial explosion of the number of plugin configurations, which in practice is often compensated only by manual and ad-hoc approaches, as reported for the Eclipse project [32].…”

Architecture challenges for internal software ecosystems: a large-scale industry case study

“…For example, Hanssen [14] studies a software product line organization and its transition towards a software ecosystem. Greiler et al [12] conduct a study of testing practices for the Eclipse ecosystem. They identify the need to involve ecosystem partners in platform testing, which is in line with our findings 3.3 and 5.3.…”