Improving Learning-from-Crowds through Expert Validation

Liu, Mengchen; Liu, Jiang; Liu, Junlin; Wang, Xiting; Zhu, Jun

doi:10.24963/ijcai.2017/324

Cited by 21 publications

(15 citation statements)

References 15 publications

(7 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Another solution is to utilize a crowd-sourcing platform in order to collect multiple annotations for each pair of connections. Accordingly, an interesting avenue of potential future work is to leverage a crowdsourcing model such as M 3 V [205] to infer the correct connection from noisy crowd-sourced labels.…”

Section: Limitationsmentioning

confidence: 99%

Bridging Text Visualization and Mining: A Task-Driven Survey

Liu

Wang

Collins

et al. 2019

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

Visual text analytics has recently emerged as one of the most prominent topics in both academic research and the commercial world. To provide an overview of the relevant techniques and analysis tasks, as well as the relationships between them, we comprehensively analyzed 263 visualization papers and 4,346 mining papers published between 1992-2017 in two fields: visualization and text mining. From the analysis, we derived around 300 concepts (visualization techniques, mining techniques, and analysis tasks) and built a taxonomy for each type of concept. The co-occurrence relationships between the concepts were also extracted. Our research can be used as a stepping-stone for other researchers to 1) understand a common set of concepts used in this research topic; 2) facilitate the exploration of the relationships between visualization techniques, mining techniques, and analysis tasks; 3) understand the current practice in developing visual text analytics tools; 4) seek potential research opportunities by narrowing the gulf between visualization and mining techniques based on the analysis tasks; and 5) analyze other interdisciplinary research areas in a similar way. We have also contributed a web-based visualization tool for analyzing and understanding research trends and opportunities in visual text analytics.

show abstract

Section: Limitationsmentioning

confidence: 99%

Bridging Text Visualization and Mining: A Task-Driven Survey

Liu

Wang

Collins

et al. 2019

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Learning from the crowd has been widely used in many different areas, e.g., web security [6,40], spam detection [53,5], phishing detection [31] and fake online account detection [13]. Moreover, using expert knowledge, such as the one provided by fact checking organizations, has been used to improve the quality and reliability of crowd learning procedures [24,18]. However, to the best of our knowledge, the present work is the first that leverages both the crowd and expert knowledge in the context of detecting and preventing the spread of fake news and misinformation.…”

Section: Related Workmentioning

confidence: 99%

Leveraging the Crowd to Detect and Reduce the Spread of Fake News and Misinformation

Kim

Tabibian

et al. 2018

Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining

182

View full text Add to dashboard Cite

Online social networking sites are experimenting with the following crowd-powered procedure to reduce the spread of fake news and misinformation: whenever a user is exposed to a story through her feed, she can flag the story as misinformation and, if the story receives enough flags, it is sent to a trusted third party for fact checking. If this party identifies the story as misinformation, it is marked as disputed. However, given the uncertain number of exposures, the high cost of fact checking, and the trade-off between flags and exposures, the above mentioned procedure requires careful reasoning and smart algorithms which, to the best of our knowledge, do not exist to date.In this paper, we first introduce a flexible representation of the above procedure using the framework of marked temporal point processes. Then, we develop a scalable online algorithm, Curb, to select which stories to send for fact checking and when to do so to efficiently reduce the spread of misinformation with provable guarantees. In doing so, we need to solve a novel stochastic optimal control problem for stochastic differential equations with jumps, which is of independent interest. Experiments on two real-world datasets gathered from Twitter and Weibo show that our algorithm may be able to effectively reduce the spread of fake news and misinformation. * This work was done during Jooyeon Kim's internship at the Max Planck Institute for Software Systems. 1 https://www.washingtonpost.com/posteverything/wp/2016/06/16/why-the-post-truth-political-era-might-be-around-for-a-while/ 2 https://www.theguardian.com/commentisfree/2016/may/13/boris-johnson-donald-trump-post-truth-politician 3 https://newsroom.fb.com/news/2016/12/news-feed-fyi-addressing-hoaxes-and-fake-news/ 4 https://www.washingtonpost.com/news/the-switch/wp/2017/06/29/twitter-is-looking-for-ways-to-let-users-flag-fake-news/ 5 their feeds, they have a choice to flag the story as misinformation and, if the story receives enough flags, it is directed to a coalition of independent organizations 6 , signatories of Poynter's International Fact Checking Code of Principles 7 , for fact checking. If the fact checking organizations identify a story as misinformation, it gets flagged as disputed and may also appear lower in the users' feeds, reducing the number of people who are exposed to misinformation. In this context, online social networking sites are giving advice to its millions of users on how to spot misinformation online 8 . However, the above mentioned procedure requires careful reasoning and smart algorithms which, to the best of our knowledge, are nonexistent to date:-Uncertain number of exposures: the spread of information over social networking sites is a stochastic process, which may depend on, e.g., the information content, the users' influence and the network structure. Thus, the number of users exposed to different stories varies greatly and we need to consider probabilistic exposure models to capture this uncertainty.-Fact checking is costly: given the myriad of (fake) sto...

show abstract

“…A recent study (Bonald and Combes 2017) shows that if the reliability of a small portion of workers can be known, the reliability of all workers can be accurately inferred, and the lower bound on the minimax estimation error can be calculated. Liu et al (2017) introduced a semi-supervised learning algorithm that selects the most informative instances and maximizes the influence of expert labels injected. They developed a complete uncertainty assessment for instance selection.…”

Section: Information Injection Methodsmentioning

confidence: 99%

Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions

Sheng

Zhang

2019

AAAI

View full text Add to dashboard Cite

With crowdsourcing systems, labels can be obtained with low cost, which facilitates the creation of training sets for prediction model learning. However, the labels obtained from crowdsourcing are often imperfect, which brings great challenges in model learning. Since 2008, the machine learning community has noticed the great opportunities brought by crowdsourcing and has developed a large number of techniques to deal with inaccuracy, randomness, and uncertainty issues when learning with crowdsourcing. This paper summarizes the technical progress in this field during past eleven years. We focus on two fundamental issues: the data (label) quality and the prediction model quality. For data quality, we summarize ground truth inference methods and some machine learning based methods to further improve data quality. For the prediction model quality, we summarize several learning paradigms developed under the crowdsourcing scenario. Finally, we further discuss several promising future research directions to attract researchers to make contributions in crowdsourcing.

show abstract

Improving Learning-from-Crowds through Expert Validation

Cited by 21 publications

References 15 publications

Bridging Text Visualization and Mining: A Task-Driven Survey

Bridging Text Visualization and Mining: A Task-Driven Survey

Leveraging the Crowd to Detect and Reduce the Spread of Fake News and Misinformation

Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions

Contact Info

Product

Resources

About