Validating metric thresholds with developers: An early result

Oliveira, Paloma; Valente, Marco; Bergel, Alexandre; Serebrenik, Alexander

doi:10.1109/icsm.2015.7332511

Cited by 3 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This table shows the percentage of classes in each repository that follow the proposed relative thresholds. For instance, the relative threshold for NOM is [80,16] and we can observe that 95% of the classes in Storm have 16 methods or less, i.e., Storm respects the relative threshold for NOM. We can also observe that only libGDX does not follow the relative threshold proposed to NOM.…”

Section: Resultsmentioning

confidence: 91%

“…Since penalty 1 = 0 (due to the 100% of compliance), we have that ComplianceRatePenalty [85,17] = 0.89. As can be observed in Figure 3.3, ComplianceRateP enalty returns zero for the following pairs [p, k]: [75,7], [75,8], [75,9], [80,8], [80,9]. Based on our tiebreaker criteria, we select the result with the highest p and then the one with the lowest k, i.e., [80,8], which leads to the following relative threshold: 80% of the classes should have NOA ≤ 8…”

Section: Illustrative Examplementioning

confidence: 99%

See 1 more Smart Citation

Extracting relative thresholds for source code metrics

Oliveira¹,

Valente²,

Lima³

2014

2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE)

Self Cite

View full text Add to dashboard Cite

Meaningful thresholds are needed for promoting software metrics as an effective instrument to measure the internal quality of systems. To address this challenge, in this PhD Thesis, we propose the concept of Relative Thresholds (RT) for evaluating metrics data following heavy-tailed distributions. The proposed thresholds assume that metric thresholds should be followed by most entities, but that it is also natural to have a number of entities in the "long-tail" that do not follow the defined limits. We describe an empirical method for deriving RT from a corpus of systems. We also perform an extensive analysis of RT: (i) we apply RT on a sample of 308 GitHub repositories. We found that most repositories follow the extracted RT; (ii) We compare the proposed RT with thresholds extracted according to a method used by the software industry. We concluded that both methods convey similar information. However, our method derives RT that can be automatically used to detect noncompliant systems; (iii) we evaluate the influence of context in our results and we concluded that the impact on RT of context changes is limited; (iv) we perform a historical analysis to check whether the proposed RT are followed by different versions of the systems under analysis. We found that our RT capture enduring software properties; (v) we check the importance of classes that do not follow the upper limit of a RT and we found these classes are important in terms of maintenance activities; (vi) we investigate the relation between the presence of bad smells in a system and its adherence to the proposed RT. We do not found evidence that noncompliant systems have more density of bad smells; (vii) we evaluated the dispersion of the metric values in the systems respecting the proposed RT, using the Gini coefficient. We found that there are different distributions of methods per class among the systems that follow the proposed RT; and (viii) We conducted a qualitative study to evaluate our method with developers. The results indicate that well-designed systems respect the RT. In contrast, we observed that developers usually have difficulties to indicate poorly-designed systems.

show abstract

Section: Resultsmentioning

confidence: 91%

Section: Illustrative Examplementioning

confidence: 99%

Extracting relative thresholds for source code metrics

Oliveira¹,

Valente²,

Lima³

2014

2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Based on this assumption, various unsupervised methods have been proposed to determine the metric thresholds according to different threshold criteria, such as Alves et al's ranking method, 6 Ferreira et al's method, 7 Vale et al's method, 8 and Oliveira et al's method. 9 For example, in the study of Alves et al, 6 the measurement data for different software systems are pooled and aggregated, and the threshold's derivation method investigates a reasonable percentage of the source code volume and the metric's variability between systems. The segmented thresholds are derived by choosing the percentage, such as 70%, 80%, and 90%, of the overall code.…”

Section: Metric Thresholds In Defect Predictionmentioning

confidence: 99%

“…Specifically, the unsupervised learning methods, which in general start from the minimum value of each metric and search for a candidate threshold toward the large value, assume that the larger the value of each metric is, the more likely the class is to be defective, that is, each metric is positively correlated with defect‐proneness. Based on this assumption, various unsupervised methods have been proposed to determine the metric thresholds according to different threshold criteria, such as Alves et al's ranking method, 6 Ferreira et al's method, 7 Vale et al's method, 8 and Oliveira et al's method 9 . For example, in the study of Alves et al, 6 the measurement data for different software systems are pooled and aggregated, and the threshold's derivation method investigates a reasonable percentage of the source code volume and the metric's variability between systems.…”

Section: Related Workmentioning

confidence: 99%

Cross‐version defect prediction using threshold‐based active learning

Mei

Liu

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

Because defects in software modules (e.g., classes) might lead to product failure and financial loss, software defect prediction enables us to better understand and control software quality. Software development is a dynamic evolutionary process that may result in data distributions (e.g., defect characteristics) varying from version to version. In this case, effective cross‐version defect prediction (CVDP) is not easy to achieve. In this paper, we aim to investigate whether the defect prediction method of the threshold‐based active learning (TAL) can tackle the problem of the different data distribution between successive versions. Our TAL method includes two stages. At the active learning stage, a committee of investigated metrics is constructed to vote on the unlabeled modules of the current version. We pick up the unlabeled module with the median of voting scores to domain experts. The domain experts test and label the selected unlabeled module. Then, we merge the selected labeled module and the remaining modules with pseudo‐labels from the current version into the labeled modules of the prior version to form enhanced training data. Based on the training data, we derive the metric thresholds used for the next iteration. At the defect prediction stage, the iterations stop when a predefined threshold is reached. Finally, we use the cutoff threshold of voting scores, that is, 50%, to predict the defect‐prone of the remaining unlabeled modules. We evaluate the TAL method on 31 versions of 10 projects with three prevalent performance indicators. The results show that TAL outperforms the baseline methods, including three variations methods, two common supervised methods, and the state‐of‐the‐art method Hybrid Active Learning and Kernel PCA (HALKP). The results indicate that TAL can effectively address the different data distribution between successive versions. Furthermore, to keep the cost of extensive testing low in practice, selecting 5% of candidate modules from the current version is sufficient for TAL to achieve a good performance of defect prediction.

show abstract

“…There are several solutions to calculate software metrics thresholds proposed in the literature [1], [18], [17], [8], [4], [23], [15], [13], [14], [5]. We summarize their characteristics and limitations in Section II.…”

Section: Introductionmentioning

confidence: 99%

A systematic process to define expert-driven software metrics thresholds (S)

Saraiva¹,

Perkusich²,

Almeida³

et al. 2019

International Conferences on Software Engineering and Knowledge Engineering

View full text Add to dashboard Cite

Software metrics are usually used for quantification, not giving the necessary support for decision making. To increase their usefulness, it is necessary to give them meaning through the definition of significant thresholds. Despite its importance, the state of the art on threshold derivation is mostly based on data-driven approaches. This paper presents a systematic approach to define thresholds for metrics in the absence of data and based on eliciting knowledge from experts. The proposed approach is based on identifying context factors that influence the thresholds for a given metric and is supported by fuzzy logic concepts to model the crisp value (i.e., collected data) into a linguistic variable (i.e., interpreted information). We present context factors elicited from three experts for the metrics code coverage, static code analysis warnings count and defect count. Further, we present cases on how to implement the proposed approach. As a result, we conclude that the approach is promising.

show abstract

Validating metric thresholds with developers: An early result

Cited by 3 publications

References 12 publications

Extracting relative thresholds for source code metrics

Extracting relative thresholds for source code metrics

Cross‐version defect prediction using threshold‐based active learning

A systematic process to define expert-driven software metrics thresholds (S)

Contact Info

Product

Resources

About