Automatic vs. Crowdsourced Sentiment Analysis

Borromeo, Ria Mae; Toyama, Motomichi

doi:10.1145/2790755.2790761

Cited by 18 publications

(17 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Crowd cost is the amount paid to the platform and the crowd workers; completion time refers to the time from when the project was launched to the time when all required responses were received, and accuracy is the degree of similarity of the results compared to a gold standard. We use the manual evaluation of the same comments in [61] as the gold standard to measure the method's accuracy. As presented in Table 1, the paid method completed significantly faster than the unpaid version while the accuracy of the two crowdsourced methods is marginally similar.…”

Section: Resultsmentioning

confidence: 99%

An investigation of unpaid crowdsourcing

Borromeo

Toyama

2016

Hum. Cent. Comput. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

The continual advancement of internet technologies has led to the evolution of how individuals and organizations operate. For example, through the internet, we can now tap a remote workforce to help us accomplish certain tasks, a phenomenon called crowdsourcing. Crowdsourcing is an approach that relies on people to perform activities that are costly or time-consuming using traditional methods. Depending on the incentive given to the crowd workers, crowdsourcing can be classified as paid or unpaid. In paid crowdsourcing, the workers are incentivized financially, enabling the formation of a robust workforce, which allows fast completion of tasks. Consequently, in unpaid crowdsourcing, the lack of financial incentive potentially leads to an unpredictable workforce and indeterminable task completion time. However, since payment to workers is not necessary, it can be an economical alternative for individuals and organizations who are more concerned about the budget than the task turnaround time. In this study, we explore unpaid crowdsourcing by reviewing crowdsourcing applications where the crowd comes from a pool of volunteers. We also evaluate its performance in sentiment analysis and data extraction projects. Our findings suggest that for such tasks, unpaid crowdsourcing completes slower but yields results of similar or higher quality compared to its paid counterpart.

show abstract

Section: Resultsmentioning

confidence: 99%

An investigation of unpaid crowdsourcing

Borromeo

Toyama

2016

Hum. Cent. Comput. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Free typing loose associations is just a lot easier than making a decision about the degree of match to a predefined category (especially hierarchical ones). It's like 90% of the value of a proper taxonomy but 10 times simpler" (Bishr & Kuhn 2007, p18 (Borromeo et al 2015). Machine learning classifiers still need training for different events as well as validation (Palen et al 2012;Cobb et al 2013), so expectations of greater responsiveness or accuracy from machines alone may be unrealistic.…”

Section: Discussionmentioning

confidence: 99%

A Case Study of Crowdsourcing Imagery Coding in Natural Disasters

Calyx¹

2017

Preprint

View full text Add to dashboard Cite

Crowdsourcing and open licensing allow more people to participate in research and humanitarian activities. Open data, such as geographic information shared through OpenStreetMap and image datasets from disasters, can be useful for disaster response and recovery work. This chapter shares a real-world case study of humanitarian-driven imagery analysis, using open-source crowdsourcing technology. Shared philosophies in open technologies and digital humanities, including remixing and the wisdom of the crowd, are reflected in this case study.

show abstract

“…The second core part in sentiment analysis, is sentiment classification -a classification that occurs at phrase/sentence/submission level, and is usually based on the aggregation of the term's labelled emotions. As with lexicon acquisition, the classification task can be automated [24,25,28,60,17,59] or performed manually [29,43,26,13].…”

Section: Introductionmentioning

confidence: 99%

“…Manually labelled classification can achieve high accuracy, but it requires additional resources, and is not easily scalable. On the other hand, automated processes are scalable but have lower accuracy [13,27].…”

Section: Introductionmentioning

confidence: 99%

A Multivalued Emotion Lexicon Created and Evaluated by the Crowd

Haralabopoulos

Wagner

McAuley

et al. 2018

2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS)

View full text Add to dashboard Cite

Sentiment analysis aims to uncover emotions conveyed through information. In its simplest form, it is performed on a polarity basis, where the goal is to classify information with positive or negative emotion. Recent research has explored more nuanced ways to perform emotion analysis. Unsupervised emotion analysis methods require a critical resource: a lexicon that is appropriate for the task at hand, in terms of the emotional range and diversity captured. Emotion analysis lexicons are created manually by domain experts and usually assign one single emotion to each word. We propose an automated workflow for creating and evaluating a multi-valued emotion lexicon created and evaluated through crowdsourcing. We compare the obtained lexicon with established lexicons and appoint expert English Linguists to assess crowd peer-evaluations. The proposed workflow provides a quality lexicon and can be used in a range of text property association tasks.

show abstract

Automatic vs. Crowdsourced Sentiment Analysis

Cited by 18 publications

References 7 publications

An investigation of unpaid crowdsourcing

An investigation of unpaid crowdsourcing

A Case Study of Crowdsourcing Imagery Coding in Natural Disasters

A Multivalued Emotion Lexicon Created and Evaluated by the Crowd

Contact Info

Product

Resources

About