The challenges of staying together while moving fast

Rubin, Julia; Rinard, Martin

doi:10.1145/2884781.2884871

Cited by 40 publications

(23 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The number of professional developers included in this study is larger or similar to those considered in many prior work, e.g., [25,28,60]. Admittedly, it is possible that results differ for other groups of developers.…”

Section: Threats To Validitymentioning

confidence: 97%

On Reliability of Patch Correctness Assessment

Bao

et al. 2019

2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

Current state-of-the-art automatic software repair (ASR) techniques rely heavily on incomplete specifications, e.g., test suites, to generate repairs. This, however, may render ASR tools to generate incorrect repairs that do not generalize. To assess patch correctness, researchers have been following two typical ways separately: (1) Automated annotation, wherein patches are automatically labeled by an independent test suite (ITS) -a patch passing the ITS is regarded as correct or generalizable, and incorrect otherwise, (2) Author annotation, wherein authors of ASR techniques annotate correctness labels of patches generated by their and competing tools by themselves. While automated annotation fails to prove that a patch is actually correct, author annotation is prone to subjectivity. This concern has caused an on-going debate on appropriate ways to assess the effectiveness of numerous ASR techniques proposed recently.To address this concern, we propose to assess reliability of author and automated annotations on patch correctness assessment. We do this by first constructing a gold set of correctness labels for 189 randomly selected patches generated by 8 state-of-the-art ASR techniques through a user study involving 35 professional developers as independent annotators. By measuring inter-rater agreement as a proxy for annotation quality -as commonly done in the literature -we demonstrate that our constructed gold set is on par with other high-quality gold sets. We then compare labels generated by author and automated annotations with this gold set to assess reliability of the patch assessment methodologies. We subsequently report several findings and highlight implications for future studies.

show abstract

Section: Threats To Validitymentioning

confidence: 97%

On Reliability of Patch Correctness Assessment

Bao

et al. 2019

2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

show abstract

“…What is, hence, required are more studies that provide empirical evidence on industrial practice in variability management [Berger et al 2013] and end-user challenges. Here, an industrial context could open up various new commercial perspectives, such as the pressure to get the next product to the market; problems to find the right expert in a large organization; and the challenge of communicating and collaborating within multidisciplinary teams [Rubin and Rinard 2016]. On a "meta" level, when considering the research approaches adopted in the surveyed studies, we observed a wide variation in quality, for example, in terms of: having an explicit research methodology; having clear research questions and evaluation criteria; supporting the replicability of the work; providing a comparison against existing work; and discussing limitations and threats to validity.…”

Section: Discussionmentioning

confidence: 99%

CASE Tool Support for Variability Management in Software Product Lines

et al. 2017

View full text Add to dashboard Cite

Software product lines (SPL) aim at reducing time-to-market and increasing software quality through extensive, planned reuse of artifacts. An essential activity in SPL is variability management, i.e., defining and managing commonality and variability among member products. Due to the large scale and complexity of today's software-intensive systems, variability management has become increasingly complex to conduct. Accordingly, tool support for variability management has been gathering increasing momentum over the last few years and can be considered a key success factor for developing and maintaining SPLs.While several studies have already been conducted on variability management, none of these analyzed the available tool support in detail. In this work, we report on a survey in which we analyzed 37 existing variability management tools identified using a systematic literature review to understand the tools' characteristics, maturity, and the challenges in the field. We conclude that while most studies on variability management tools provide a good motivation and description of the research context and challenges, they often lack empirical data to support their claims and findings. It was also found that quality attributes important for the practical use of tools such as usability, integration, scalability, and performance were out of scope for most studies. CCS Concepts: • General and reference → Surveys and overviews • Software and its engineering → Software product lines • Software and its engineering → Software notations and tools • Software and its engineering → Software configuration management and version control systems XX:2 • R. Bashroush et al.Defining and managing commonalities and variability in software product lines is widely referred to as variability management and is a key step of the SPL engineering process [van Gurp et al. 2001]. The variability management process guides the construction of product line variability models. Different types of variability models have been proposed, e.g., feature models, decision models, Orthogonal Variability Models (OVM), and UML-based approaches. In Section 1.1 we provide an overview of existing modeling approaches. For a detailed comparison and classification of variability modeling approaches we refer to [Czarnecki et al. 2012] and [Sinnema and Deelstra 2007]. Variability models define the commonalities and variability of the product line from a problem space (e.g., features, decisions, or variation points) and a solution space (e.g., the reusable assets or variants) perspective along with the relationships that exist between these two spaces and among the elements in these spaces. Example relationships include exclusivity (when two features cannot exist in one product at the same time); inclusivity (when the existence of one feature depends on another); and alternatives (when only one of a number of alternative features can be supported), to name a few. Variability models tend to be very large in size, in many cases comprising thousands of features, and comp...

show abstract

“…Although software quality has always been a common topic in software engineering [5,35], most of the studies have focused on pre-release data, primarily due to the developers' concern about finding the appropriate balance between the amount of testing required and the quality of software (e.g. [62,12]). There have been a number of works on predicting and improving the software quality as well (e.g.…”

Section: Related Workmentioning

confidence: 99%

Deriving a usage-independent software quality metric

Dey¹,

Mockus²

2020

Empir Software Eng

View full text Add to dashboard Cite

Context: The extent of post-release use of software affects the number of faults, thus biasing quality metrics and adversely affecting associated decisions. The proprietary nature of usage data limited deeper exploration of this subject in the past. Objective: To determine how software faults and software use are related and how, based on that, an accurate quality measure can be designed. Method: Via Google Analytics we measure new users, usage intensity, usage frequency, exceptions, and release date and duration for complex proprietary mobile applications for Android and iOS. We utilize Bayesian Network and Random Forest models to explain the interrelationships and to derive the usage independent release quality measure. To increase external validity, we also investigate the interrelationship among various code complexity measures, usage (downloads), and number of issues for 520 NPM packages. We derived a usage-independent quality measure from these analyses, and applied it on 4430 popular NPM packages to construct timelines for comparing the perceived quality (number of issues) and our derived measure of quality during the lifetime of these packages. Results: We found the number of new users to be the primary factor determining the number of exceptions, and found no direct link between the intensity and frequency of software usage and software faults. Crashes increased with the power of 1.02-1.04 of new user for the Android app and power of 1.6 for the iOS app. Release quality expressed as crashes per user was independent of other usage-related predictors, thus serving as a usage independent measure of software quality. Usage also affected quality in NPM, where downloads were strongly associated with numbers of issues, even after taking the other code complexity measures into consideration. Unlike in mobile case where exceptions per user decrease over time, for 45.8% of the NPM packages the number of issues per download increase. Conclusions: We expect our result and our proposed quality measure will help gauge release quality of a software more accurately and inspire further research in this area.

show abstract

The challenges of staying together while moving fast

Cited by 40 publications

References 57 publications

On Reliability of Patch Correctness Assessment

On Reliability of Patch Correctness Assessment

CASE Tool Support for Variability Management in Software Product Lines

Deriving a usage-independent software quality metric

Contact Info

Product

Resources

About