Cheap and fast---but is it good?

Snow, Rion; O’Connor, Brendan; Jurafsky, Daniel; Ng, Andrew Y.

doi:10.3115/1613715.1613751

Cited by 1,022 publications

(142 citation statements)

References 24 publications

Supporting

Mentioning

141

Contrasting

Unclassified

Order By: Relevance

“…To obtain labeled training data for the classifier, we utilized workers from the Amazon Mechanical Turk 7 . It has been shown that manual labeling from Amazon Turk can correlate well with experts [11]. We used thousands of workers to assign sentiments for a large random sample of tweets, ensuring that each tweet was labeled by three different people.…”

Section: Sentiment Analysismentioning

confidence: 99%

How does market concern derived from the Internet affect oil prices?

Guo

2013

Applied Energy

View full text Add to dashboard Cite

Abstract-In recent years, social media has become ubiquitous and important for social networking and content sharing. And yet, the content that is generated from these websites remains largely untapped. In this paper, we demonstrate how social media content can be used to predict real-world outcomes. In particular, we use the chatter from Twitter.com to forecast box-office revenues for movies. We show that a simple model built from the rate at which tweets are created about particular topics can outperform market-based predictors. We further demonstrate how sentiments extracted from Twitter can be further utilized to improve the forecasting power of social media.

show abstract

Section: Sentiment Analysismentioning

confidence: 99%

How does market concern derived from the Internet affect oil prices?

Guo

2013

Applied Energy

View full text Add to dashboard Cite

show abstract

“…In [9], a high agreement between MTurk non-expert annotations and existing gold-standard labels provided by expert labelers for five natural language processing tasks is demonstrated. Multiple labeling has also been shown to be useful for improving the data quality of annotations of non-experts [5].…”

Section: Related Workmentioning

confidence: 88%

“…Earlier work on inferring ground truth from subjective labels includes Smyth et al [6], Sheng et al [5], and Snow et al [9]. In [6], the latent relation between subjective labels and true labels by EM algorithm was studied and it was shown that the posterior conditional probabilities of subjective labels and true labels generally agree with intuition and often (70% of the samples) correspond to a majority vote among the labelers.…”

Section: Related Workmentioning

confidence: 99%

Collecting high quality overlapping labels at low cost

Yang

Mityagin

Svore

et al. 2010

Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

This paper studies quality of human labels used to train search engines' rankers. Our specific focus is performance improvements obtained by using overlapping relevance labels, which collecting multiple human judgments for each training sample. The paper explores whether, when, and for which should obtain overlapping training labels, as well as labels per sample are needed. The proposed scheme collects additional labels only for a subset of training samples, specifically for those that are labeled relevant by a Our experiments show that this labeling schem NDCG of two Web search rankers on several real with a low labeling overhead of around 1.4 labels per sample This labeling scheme also outperforms several overlapping labels, such as simple k-overlap, majority vote, the highest labels, etc. Finally, the paper presents a study of how many overlapping labels are needed to get the best in retrieval accuracy.

show abstract

“…With the increasing popularity, a set of crowdsourcing market platforms such as Amazon Mechanical Turk and CrowdFlower [5] have emerged, which enable human workers to perform tasks on the Internet. Crowdsourcing applications have been adopted in a wide range of applications such as image search [24], natural language annotations [22] and information retrieval [11]. Moreover, crowdsourcing has also been incorporated into database design and relational query processing [10,12,18].…”

Section: Related Workmentioning

confidence: 99%

Maximizing the number of worker's self-selected tasks in spatial crowdsourcing

Deng

Shahabi

Demiryurek

2013

Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems

166

144

View full text Add to dashboard Cite

With the progress of mobile devices and wireless broadband, a new eMarket platform, termed spatial crowdsourcing is emerging, which enables workers (aka crowd) to perform a set of spatial tasks (i.e., tasks related to a geographical location and time) posted by a requester. In this paper, we study a version of the spatial crowdsourcing problem in which the workers autonomously select their tasks, called the worker selected tasks (WST) mode. Towards this end, given a worker, and a set of tasks each of which is associated with a location and an expiration time, we aim to find a schedule for the worker that maximizes the number of performed tasks. We first prove that this problem is NP-hard. Subsequently, for small number of tasks, we propose two exact algorithms based on dynamic programming and branch-and-bound strategies. Since the exact algorithms cannot scale for large number of tasks and/or limited amount of resources on mobile platforms, we also propose approximation and progressive algorithms. We conducted a thorough experimental evaluation on both real-world and synthetic data to compare the performance and accuracy of our proposed approaches.

show abstract

Cheap and fast---but is it good?

Cited by 1,022 publications

References 24 publications

How does market concern derived from the Internet affect oil prices?

How does market concern derived from the Internet affect oil prices?

Collecting high quality overlapping labels at low cost

Maximizing the number of worker's self-selected tasks in spatial crowdsourcing

Contact Info

Product

Resources

About