A Survey of the Use of Test Report in Crowdsourced Testing

Huang, Song; Chen, Hao; Hui, Zhanwei; Liu, Yuchan

doi:10.1109/qrs51102.2020.00062

Cited by 4 publications

(4 citation statements)

References 96 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, finding valuable information from an overwhelming number of crowdsourced test reports is time-consuming because of the high duplicate rate. Duplicate test reports mean the same or similar descriptions revealing the same bugs [4]. Crowdsourced test reports have a significant proportion of duplicates for three reasons:…”

Section: Background and Motivationmentioning

confidence: 99%

“…In crowdsourced testing, a large workforce concurrently performs test tasks posted on the crowdsourcing platform and submits test results in the form of test reports. Test task requesters usually obtain numerous test reports that have a high rate of duplication and poor quality [4]. How to inspect these test reports efficiently becomes an unavoidable challenge for test task requesters; therefore, they seek methods to identify and prioritize useful information from a large number of test reports to speed up the software development process.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Severity-Oriented Multi-Objective Crowdsourced Test Reports Prioritization

Wu,

Xiao

2024

Int. J. Image Grap.

View full text Add to dashboard Cite

Crowdsourced testing for mobile applications is widespread. Efficiently and swiftly inspecting enormous test reports is a crucial challenge in crowdsourced testing. Most existing methods focus on finding different bugs earlier, i.e. the diversity of bugs, while ignoring bug severity. We present a multi-objective prioritization method for crowdsourced test reports that takes into account the diversity and severity of bugs. In the beginning, we ranked the test reports according to severity, then clustered the test reports by fusing textual and image features, and finally used the clustering results to adjust the initial ranking to ensure diversity. To validate our method, we conducted experiments using an industrial crowdsourced test report dataset comprised of six mobile application projects. The results demonstrate that our method can find more different high-severity bugs earlier than existing methods.

show abstract

Section: Background and Motivationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Severity-Oriented Multi-Objective Crowdsourced Test Reports Prioritization

Wu,

Xiao

2024

Int. J. Image Grap.

View full text Add to dashboard Cite

show abstract

“…Crowdsourced testing will produce a large number of test reports which take developers a lot of time to inspect. To reduce the inspection cost of test reports, researchers have conducted extensive studies to reduce the number of inspected test reports or improve the quality of inspected test reports [33].…”

Section: Multi-bug Test Reportsmentioning

confidence: 99%

Crowdsourced Test Report Prioritization Based on Text Classification

Yang

Chen

2022

IEEE Access

View full text Add to dashboard Cite

In crowdsourced testing, crowd workers from different places help developers conduct testing and submit test reports for the observed abnormal behaviors. Developers manually inspect each test report and make an initial decision for the potential bug. However, due to the poor quality, test reports are handled extremely slowly. Meanwhile, due to the limitation of resources, some test reports are not handled at all. Therefore, some researchers attempt to resolve the problem of test report prioritization and have proposed many methods. However, these methods do not consider the impact of duplicate test reports. In this paper, we focus on the problem of test report prioritization and present a new method named DivClass by combining a diversity strategy and a classification strategy. First, we leverage Natural Language Processing (NLP) techniques to preprocess crowdsourced test reports. Then, we build a similarity matrix by introducing an asymmetric similarity computation strategy. Finally, we combine the diversity strategy and the classification strategy to determine the inspection order of test reports. To validate the effectiveness of DivClass, experiments are conducted on five crowdsourced test report datasets. Experimental results show that DivClass achieves 0.8887 in terms of APFD (Average Percentage of Fault Detected) and improves the state-of-the-art technique DivRisk by 14.12% on average. The asymmetric similarity computation strategy can improve DivClass by 4.82% in terms of APFD on average. In addition, empirical results show that DivClass can greatly reduce the number of inspected test reports. 17 18 157 of software. Screenshot contains some pictures that may 158 capture the states of software when occurring a bug. 159 Compared with traditional testing, crowdsourced testing 160 has its own characteristics. First, to attract more workers 161 for testing, test tasks are usually financially compensated.162 Many workers tend to quickly complete test tasks and submit 163 more test reports. In such a way, the number of submitted 164 test reports is large, obviously outperforming the available 165 resources of developers. Thus, many test reports are not dealt 166 with timely and some important bugs may be not detected 167 and fixed before releasing. Second, workers tend to detect 168 easily discovered errors in software rather than critical bugs 169 or bugs that are hard to reproduce. Therefore, the submitted 170 test reports are highly redundant. Inspecting duplicate test 171 reports will spend unnecessary time and resources of devel-consideration and design an asymmetric similarity computing 207 method. 208 III. METHODOLOGY 209 In this section, we describe the implementation details of 210 DivClass. As shown in Figure 1, DivClass is composed of 211 three procedures. First, we adopt NLP to preprocess crowd-212 sourced test reports and build a keyword dictionary. Then, 213 an asymmetric similarity metric is constructed by employ-214 ing the Jaccard similarity coefficient. Finally, we combine a 215 diversity strate...

show abstract

“…The problem with crowdsourced testing is dealing with the large number of test reports [3]. Although a large number of test reports are obtained, the number of valuable reports was relatively small, as duplicate reports amount to almost 82% [4].…”

Section: Introductionmentioning

confidence: 99%

Optimizing Prioritization of Crowdsourced Test Reports of Web Applications through Image-to-Text Conversion

Xiao,

Wu,

Chen

et al. 2024

Symmetry

View full text Add to dashboard Cite

Collating vast test reports is a time-consuming and laborious task in crowdsourced testing. Crowdsourced test reports are usually presented in two ways, one as text and the other as images, which have symmetrical content. Researchers have proposed many text- and image-based methods for prioritizing crowdsourced test reports of mobile applications. However, crowdsourced test reports of web applications typically have clearer textual descriptions of errors but noisier screenshots of errors. This gap motivates us to propose a method for prioritizing crowdsourced test reports of web applications to detect all errors earlier. In this paper, we integrate text and image information from test reports to enhance the analysis process. First, we use the natural language processing (NLP) technique to extract textual features of error descriptions and then symmetrically extract image features of error screenshots, i.e., we use the optical character recognition (OCR) technique to obtain textual information in the screenshots and then also use the NLP technique to extract features. To validate our approach, we conduct experiments on 717 test reports. The experimental results show that our method has a higher APFD (average percentage fault detection) and shorter runtime than state-of-the-art prioritization methods.

show abstract

A Survey of the Use of Test Report in Crowdsourced Testing

Cited by 4 publications

References 96 publications

Severity-Oriented Multi-Objective Crowdsourced Test Reports Prioritization

Severity-Oriented Multi-Objective Crowdsourced Test Reports Prioritization

Crowdsourced Test Report Prioritization Based on Text Classification

Optimizing Prioritization of Crowdsourced Test Reports of Web Applications through Image-to-Text Conversion

Contact Info

Product

Resources

About