Machine learning from crowds: A systematic review of its applications

Rodrigo, Enrique González; Aledo, Juan A.; Gámez, José A.

doi:10.1002/widm.1288

Cited by 11 publications

(9 citation statements)

References 151 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…a URL, to a wide variety of free online resources media at a low cost. Rodrigo et al (2019) opines that crowdsourcing opens the door to solving a wide variety of problems allowing us to obtain relatively low cost labelled data in a small amount of time. University libraries can explore the use of AI for crowdsourcing, which reduces the financial burden on its parent institution.…”

Section: Theoretical Modelmentioning

confidence: 99%

See 1 more Smart Citation

Artificial intelligence (AI) library services innovative conceptual framework for the digital transformation of university education

Okunlaya

Abdullah

Alias

2022

LHT

133

View full text Add to dashboard Cite

PurposeArtificial intelligence (AI) is one of the latest digital transformation (DT) technological trends the university library can use to provide library users with alternative educational services. AI can foster intelligent decisions for retrieving and sharing information for learning and research. However, extant literature confirms a low adoption rate by the university libraries in using AI to provide innovative alternative services, as this is missing in their strategic plan. The research develops (AI-LSICF) an artificial intelligence library services innovative conceptual framework to provide new insight into how AI technology can be used to deliver value-added innovative library services to achieve digital transformation. It will also encourage library and information professionals to adopt AI to complement effective service delivery.Design/methodology/approachThis study adopts a qualitative content analysis to investigate extant literature on how AI adoption fosters innovative services in various organisations. The study also used content analysis to generate possible solutions to aid AI service innovation and delivery in university libraries.FindingsThis study uses its findings to develop an Artificial Intelligence Library Services Innovative Conceptual Framework (AI-LSICF) by integrating AI applications and functions into the digital transformation framework elements and discussed using a service innovation framework.Research limitations/implicationsIn research, AI-LSICF helps increase an understanding of AI by presenting new insights into how the university library can leverage technology to actualise innovation in service provision to foster DT. This trail will be valuable to scholars and academics interested in addressing the application pathways of AI library service innovation, which is still under-explored in digital transformation.Practical implicationsIn practice, AI-LSICF could reform the information industry from its traditional brands into a more applied and resolutely customer-driven organisation. This reformation will awaken awareness of how librarians and information professionals can leverage technology to catch up with digital transformation in this age of the fourth industrial revolution.Social implicationsThe enlightenment of AI-LSICF will motivate library professionals to take advantage of AI's potential to enhance their current business model and achieve a unique competitive advantage within their community.Originality/valueAI-LSICF development serves as a revelation, motivating university libraries and information professionals to consider AI in their strategic plan to enable technology to support university education. This act will enable alternative service delivery in the face of unforeseen circumstances like technological disruption and the present global COVID-19 pandemic that requires non-physical interaction.

show abstract

Section: Theoretical Modelmentioning

confidence: 99%

“…a URL, to a wide variety of free online resources media at a low cost. Rodrigo et al. (2019) opines that crowdsourcing opens the door to solving a wide variety of problems allowing us to obtain relatively low cost labelled data in a small amount of time.…”

Section: Theoretical Modelmentioning

confidence: 99%

Artificial intelligence (AI) library services innovative conceptual framework for the digital transformation of university education

Okunlaya

Abdullah

Alias

2022

LHT

133

View full text Add to dashboard Cite

show abstract

“…In particular, crowdsourcing represents a popular way to obtain data annotations [26]. However, on crowdsourcing platforms, e.g., Amazon's Mechanical Turk [27,28], CloudResearch (formerly TurkPrime) [29], and Prolific Academic [30], multiple error-prone annotators have to be considered [31]. Otherwise, annotation mistakes (e.g., noisy class labels) will degrade the classification model's performance [32,33].…”

Section: Introductionmentioning

confidence: 99%

A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification

et al. 2021

View full text Add to dashboard Cite

Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, a wide range of novel AL strategies has been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: (1) They explicitly consider (multiple) human annotators whose performances can be affected by various factors, such as missing expertise. (2) They generalize the interaction with human annotators by considering different query and annotation types, such as asking an annotator for feedback on an inferred classification rule. (3) They take more complex cost schemes regarding annotations and misclassifications into account. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.

show abstract

“…These include the unsatisfactory inter-person consistency and the extensive efforts required to perform a crowdsourcing task. 9 ML refers to the application of arti cial intelligence (AI) that enables computer systems to learn and improve from experience, typically from large amounts of training data, without being explicitly programmed. The use of ML in the eld of text analyzing is called natural language processing (NLP), referring to the analysis of the human language.…”

Section: Introductionmentioning

confidence: 99%

Automated Screener Based on Convolutional Neural Network for Randomized Controlled Trials in Chinese Language: A Comparative Study of Different Classification Strategies

Chen

Mao

Xie

et al. 2021

Preprint

View full text Add to dashboard Cite

Objective: To explore the influence of modified literature classification strategies of Chinese biomedical literature on an automated screener based on conventional algorithm.Methods: Citations of studies indexed as ‘Oral Science’ published in Chinese between 2014 and 2018 were retrieved from the China National Knowledge Infrastructure. Apart from dividing the studies into 2 categories (RCTs and non-RCTs), 3-category (RCTs, may-be-RCTs, and non-RCTs) and 5-category (RCTs, randomization-unclear controlled trials, non-randomized clinical trials/studies, non-clinical literature, and unclear) classification were also employed. The multi-category strategies took into consideration the diversity of study types and the presence of expression vagueness. Similar to real-world practice, full-text-needed studies included those that certainly concerned RCTs and those that might be RCTs but lacked information in their abstracts. Screening and classification were performed independently by 2 experienced researchers. The classification results after peer discussion and/or senior decision were used for the training of the CNN model. The probability thresholds for the classification of each category were set at a high sensitivity level.The area under the receiver-operator curve (AUC) was calculated when applicable. An isolated sample of citations was used in a prospective comparative trial that compared the sensitivity (SEN) and specificity (SPE) of screening RCTs, may-be-RCTs, and full-text-needed studies by using algorithms with different strategies and manual screening.Results:In total, 12,166 citations were used for CNN model training. All 3 training strategies performed well in RCTs-screening with AUCs being higher than 0.99. The training exhibited that, when screening for RCTs, the 5- and 3-category strategies can yield better performance than the 2-category strategy. When screening for may-be-RCTs and full text-needed studies, the 5-category model achieved better SENs while the 3-category model achieved higher SPEs. The comparative trial with 1,422 samples presented similar results.Conclusion: The CNN algorithm has promising results in the automatic screening of Chinese literature. The multi-category training strategies considering different study types and expression vagueness are more suitable for CNN training and can help achieve better screening sensitivity and specificity.

show abstract

Machine learning from crowds: A systematic review of its applications

Cited by 11 publications

References 151 publications

Artificial intelligence (AI) library services innovative conceptual framework for the digital transformation of university education

Artificial intelligence (AI) library services innovative conceptual framework for the digital transformation of university education

A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification

Automated Screener Based on Convolutional Neural Network for Randomized Controlled Trials in Chinese Language: A Comparative Study of Different Classification Strategies

Contact Info

Product

Resources

About