Crowdsourced Test Report Prioritization Based on Text Classification

Abstract

In crowdsourced testing, crowd workers from different places help developers conduct testing and submit test reports for the observed abnormal behaviors. Developers manually inspect each test report and make an initial decision for the potential bug. However, due to the poor quality, test reports are handled extremely slowly. Meanwhile, due to the limitation of resources, some test reports are not handled at all. Therefore, some researchers attempt to resolve the problem of test report prioritization and have proposed many methods. However, these methods do not consider the impact of duplicate test reports. In this paper, we focus on the problem of test report prioritization and present a new method named DivClass by combining a diversity strategy and a classification strategy. First, we leverage Natural Language Processing (NLP) techniques to preprocess crowdsourced test reports. Then, we build a similarity matrix by introducing an asymmetric similarity computation strategy. Finally, we combine the diversity strategy and the classification strategy to determine the inspection order of test reports. To validate the effectiveness of DivClass, experiments are conducted on five crowdsourced test report datasets. Experimental results show that DivClass achieves 0.8887 in terms of APFD (Average Percentage of Fault Detected) and improves the state-of-the-art technique DivRisk by 14.12% on average. The asymmetric similarity computation strategy can improve DivClass by 4.82% in terms of APFD on average. In addition, empirical results show that DivClass can greatly reduce the number of inspected test reports. 17 18 157 of software. Screenshot contains some pictures that may 158 capture the states of software when occurring a bug. 159 Compared with traditional testing, crowdsourced testing 160 has its own characteristics. First, to attract more workers 161 for testing, test tasks are usually financially compensated.162 Many workers tend to quickly complete test tasks and submit 163 more test reports. In such a way, the number of submitted 164 test reports is large, obviously outperforming the available 165 resources of developers. Thus, many test reports are not dealt 166 with timely and some important bugs may be not detected 167 and fixed before releasing. Second, workers tend to detect 168 easily discovered errors in software rather than critical bugs 169 or bugs that are hard to reproduce. Therefore, the submitted 170 test reports are highly redundant. Inspecting duplicate test 171 reports will spend unnecessary time and resources of devel-consideration and design an asymmetric similarity computing 207 method. 208 III. METHODOLOGY 209 In this section, we describe the implementation details of 210 DivClass. As shown in Figure 1, DivClass is composed of 211 three procedures. First, we adopt NLP to preprocess crowd-212 sourced test reports and build a keyword dictionary. Then, 213 an asymmetric similarity metric is constructed by employ-214 ing the Jaccard similarity coefficient. Finally, we combine a 215 diversity strate...

A Review of Trending Crowdsourcing Topics in Software Engineering Highlighting Mobile Crowdsourcing and AI Utilization

“…Can general crowdsourced software engineering methods be used in mobile crowdsourcing? Mobile development comes with its own set of problems, especially in software testing [40]- [44] describe these issues. In this review, there are ten papers that focus exclusively on mobile crowdsourcing, specifically mobile testing.…”

“…[42] state that in addition to the large number of test reports produced by crowdsourcing that needs inspecting, one specific issue of mobile application test reports is that they have more screenshots than text descriptions of the tests. In addition to this paper, [40], [41], [43], [44] deal with these issues as well. To solve the issues of the number of reports and screenshots, the authors of [42] propose a novel method to understand text and images to cluster test reports.…”

“…The results show that this method has promising performance in prioritizing the test reports, with an average percentage of faults detected (APFD) of more than 0.8. The authors of [44] devise a new method called DivClass to prioritize test reports. DivClass combines diversity and classification strategies to order the reports for inspection.…”

“…DivClass combines diversity and classification strategies to order the reports for inspection. A feature of this method is that it handles duplicate test reports similar to the method proposed in [40], [42]- [44] in which they use natural language processing to analyze the test reports in one of their method steps. The next step in [44] is to build a similarity matrix using an asymmetric computation strategy.…”

“…A feature of this method is that it handles duplicate test reports similar to the method proposed in [40], [42]- [44] in which they use natural language processing to analyze the test reports in one of their method steps. The next step in [44] is to build a similarity matrix using an asymmetric computation strategy. The final step consists of the previous two steps to prioritize the reports.…”