FRUGAL: Unlocking SSL for Software Analytics

Abstract

Standard software analytics often involves having a large amount of data with labels in order to commission models with acceptable performance. However, prior work has shown that such requirements can be expensive, taking several weeks to label thousands of commits, and not always available when traversing new research problems and domains. Unsupervised Learning is a promising direction to learn hidden patterns within unlabelled data, which has only been extensively studied in defect prediction. Nevertheless, unsupervised learning can be ineffective by itself and has not been explored in other domains (e.g., static analysis and issue close time).Motivated by this literature gap and technical limitations, we present FRUGAL, a tuned semi-supervised method that builds on a simple optimization scheme that does not require sophisticated (e.g., deep learners) and expensive (e.g., 100% manually labelled data) methods. FRUGAL optimizes the unsupervised learner's configurations (via a simple grid search) while validating our design decision of labelling just 2.5% of the data before prediction.As shown by the experiments of this paper FRUGAL outperforms the state-of-the-art adoptable static code warning recognizer and issue closed time predictor, while reducing the cost of labelling by a factor of 40 (from 100% to 2.5%). Hence we assert that FRUGAL can save considerable effort in data labelling especially in validating prior work or researching new problems.Based on this work, we suggest that proponents of complex and expensive methods should always baseline such methods against simpler and cheaper alternatives. For instance, a semi-supervised learner like FRUGAL can serve as a baseline to the state-of-the-art software analytics. INTRODUCTIONSoftware analytics can guide improvements to software quality, maintenance and security. For example, analytics can discover which static code warnings are adoptable [81,90]; whether the new issues can be easily fixed [50,93]; where software defects are likely to occur [1,64]; which comments likely to contain technical debts [48,94]; what the current health conditions of these opensource projects [84]; or how to distinguish security bug reports [74].

Fast changeset-based bug localization with BERT

“…On the other hand, automated mining for labels is far more likely to meet the demand for data quantity, however at the cost of introducing noise in the form of both false positives and false negatives [8,53]. Hence, collecting large amount of good quality labelled data can pose a significant challenge for many important software engineering problems and tasks, in particular those that require single project data (i.e., within project) [50]. A recent approach to address this problem is to use transfer learning, i.e., pre-training a model with unsupervised learning on a large, general corpus, followed by fine-tuning via supervised learning towards the downstream task.…”