Learning deep networks with crowdsourcing for relevance evaluation

Wu, Ming Ting; Yin, Xiaochun; Li, Qianmu; Zhang, Jing; Feng, Xinqi; Cao, Qi; Shen, Haiyuan

doi:10.1186/s13638-020-01697-2

Cited by 4 publications

(4 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another method is crowdsourcing, where experts perform small tasks to address difficult problems. There are many applications of this approach, for example, a query-oriented system for data cleaning with oracle crowds [5] or a technique that improves the quality of labels by repeatedly labeling each sample and creating integrated labels [34]. Another technique is metamorphic testing, originally developed to evaluate software quality and verify relations among a group of test outputs with corresponding test inputs [36].…”

Section: Related Workmentioning

confidence: 99%

Evaluation of the Limit of Detection in Network Dataset Quality Assessment with PerQoDA

Wasielewska

Soukup

Čejka

et al. 2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Machine learning is recognised as a relevant approach to detect attacks and other anomalies in network traffic. However, there are still no suitable network datasets that would enable effective detection. On the other hand, the preparation of a network dataset is not easy due to privacy reasons but also due to the lack of tools for assessing their quality. In a previous paper, we proposed a new method for data quality assessment based on permutation testing. This paper presents a parallel study on the limits of detection of such an approach. We focus on the problem of network flow classification and use well-known machine learning techniques. The experiments were performed using publicly available network datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Evaluation of the Limit of Detection in Network Dataset Quality Assessment with PerQoDA

Wasielewska

Soukup

Čejka

et al. 2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

show abstract

“…Among various types of CAPTCHA, image annotation ones could capture a remarkable attention so that the leader in the field, reCAPTCHA, is leveraging this mechanism. From another perspective, the need for annotated images is not a decreasing one, since, as an example, machine (deep) learning algorithms require such knowledge for the training phase and/or other tasks (Vaughan, 2017;Russakovsky et al, 2015;Wu et al, 2020) as well as ground truth to evaluate models. Moreover, the CAPTCHA-based image annotation mechanics can be inspirational for designing innovative image annotation and segmentation frameworks, such as the recent work introduced in (Dang et al, 2022), Vessel-CAPTCHA, aimed to segment and annotate 3D brain vessel images by using a CAPTCHA-like mechanism.…”

Section: Research Motivationsmentioning

confidence: 99%

“…From another perspective, the need for annotated images is not a decreasing one, since, as an example, machine (deep) learning algorithms require such knowledge for the training phase and/or other tasks (Vaughan, 2017; Russakovsky et al. , 2015; Wu et al. , 2020) as well as ground truth to evaluate models.…”

Section: Research Motivationsmentioning

confidence: 99%

CAPTCHA for crowdsourced image annotation: directions and efficiency analysis

Moradi

Keyvanpour

2022

AJIM

View full text Add to dashboard Cite

Purpose Image annotation plays an important role in image retrieval process, especially when it comes to content-based image retrieval. In order to compensate the intrinsic weakness of machines in performing cognitive task of (human-like) image annotation, leveraging humans’ knowledge and abilities in the form of crowdsourcing-based annotation have gained momentum. Among various approaches for this purpose, an innovative one is integrating the annotation process into the CAPTCHA workflow. In this paper, the current state of the research works in the field and experimental efficiency analysis of this approach are investigated.Design/methodology/approach At first, and with the aim of presenting a current state report of research studies in the field, a comprehensive literature review is provided. Then, several experiments and statistical analyses are conducted to investigate how CAPTCHA-based image annotation is reliable, accurate and efficient.Findings In addition to study of current trends and best practices for CAPTCHA-based image annotation, the experimental results demonstrated that despite some intrinsic limitations on leveraging the CAPTCHA as a crowdsourcing platform, when the challenge, i.e. annotation task, is selected and designed appropriately, the efficiency of CAPTCHA-based image annotation can outperform traditional approaches. Nonetheless, there are several design considerations that should be taken into account when the CAPTCHA is used as an image annotation platform.Originality/value To the best of the authors’ knowledge, this is the first study to analyze different aspects of the titular topic through exploration of the literature and experimental investigation. Therefore, it is anticipated that the outcomes of this study can draw a roadmap for not only CAPTCHA-based image annotation but also CAPTCHA-mediated crowdsourcing and even image annotation.

show abstract

“…It is trained on highly configured computers by using a large number of data labels, which are not only fast to test but also perform well and are easy to deploy and apply [48]. However, the relationship between different datasets in deep learning methods and different network architecture designs and network generalization capabilities is still being explored by a large number of researchers to find out what the essence [49][50][51]. For different data objects, such as apple images, the relationships between how much data to use, what type of data to use, and which network architecture to use will have acceptable generalization capabilities are still unclear, and their interpretability needs further research in progress.…”

Section: Further Research Perspectivesmentioning

confidence: 99%

Multi-Feature Patch-Based Segmentation Technique in the Gray-Centered RGB Color Space for Improved Apple Target Recognition

et al. 2021

View full text Add to dashboard Cite

In the vision system of apple-picking robots, the main challenge is to rapidly and accurately identify the apple targets with varying halation and shadows on their surfaces. To solve this problem, this study proposes a novel, multi-feature, patch-based apple image segmentation technique using the gray-centered red–green–blue (RGB) color space. The developed method presents a multi-feature selection process, which eliminates the effect of halation and shadows in apple images. By exploring all the features of the image, including halation and shadows, in the gray-centered RGB color space, the proposed algorithm, which is a generalization of K-means clustering algorithm, provides an efficient target segmentation result. The proposed method is tested on 240 apple images. It offered an average accuracy rate of 98.79%, a recall rate of 99.91%, an F1 measure of 99.35%, a false positive rate of 0.04%, and a false negative rate of 1.18%. Compared with the classical segmentation methods and conventional clustering algorithms, as well as the popular deep-learning segmentation algorithms, the proposed method can perform with high efficiency and accuracy to guide robotic harvesting.

show abstract

Learning deep networks with crowdsourcing for relevance evaluation

Cited by 4 publications

References 39 publications

Evaluation of the Limit of Detection in Network Dataset Quality Assessment with PerQoDA

Evaluation of the Limit of Detection in Network Dataset Quality Assessment with PerQoDA

CAPTCHA for crowdsourced image annotation: directions and efficiency analysis

Multi-Feature Patch-Based Segmentation Technique in the Gray-Centered RGB Color Space for Improved Apple Target Recognition

Contact Info

Product

Resources

About