Testing in the Wild: The Social and Organisational Dimensions of Real World Practice

Abstract

Testing is a key part of any systems engineering project. There is an extensive literature on testing, but very little that focuses on how testing is carried out in real-world circumstances. This is partly because current practices are often seen as unsophisticated and ineffective. We believe that by investigating and characterising the real-world work of testing we can help question why such 'bad practices' occur and how improvements might be made. We also argue that the testing literature is too focused on technological issues when many of the problems, and indeed strengths, have as much do with work and organisation. In this paper we use empirical examples from four systems engineering projects to demonstrate how and in what ways testing is a cooperative activity. In particular we demonstrate the ways in which testing is situated within organisational work and satisfices organisational and marketplace demands.

When, how, and why developers (do not) test in their IDEs

“…By comparing the state of the practice to the state of the art of testing in the IDE [8][9][10], we aim to understand the testing patterns and needs of software engineers, expressed in our five research questions: If we study these research questions in a maximally large and varied population of software engineers, the answers to them can provide important implications for practitioners, designers of nextgeneration IDEs, and researchers. To this end, we have set up an open-ended, longitudinal field study [11] that has run for five months and involved 416 software engineers from industry as well as open-source projects around the world.…”

“…These tools include Syde [32], Spyware [33], CodingTracker [34], the "Change-Oriented Programming Environment," 8 the "Eclipse Usage Data Collector," 9 QuantifiedDev, 10 Codealike, 11 and the work by Minelli et al [35]. However, none of these focuses on time-related developer testing.…”

From Artefacts to Infrastructures

“…This sentiment is manifest in the CSCW emphasis on workplace studies and especially single site-implementations (in a specific department or an organisation) (see for instance Fitzpatrick 2003), the influence of ethnographically inspired research methods (Rooksby et al 2009), a focus on the adoption of a given system rather than on how users collaborate using multiple systems (Berg 1999), and studies of relatively small numbers of users (tens or hundreds rather than thousands).…”

Dynamic Prioritization in Regression Testing

“…They also mention that the focus in industry is more on automation to decrease the cost and effort of regression testing. Rooksby et al [6] argue for the need to investigate and characterize the real-world work of testing.…”