Crowdsourcing services, such as Amazon Mechanical Turk, allow for easy distribution of small tasks to a large number of workers. Unfortunately, since manually verifying the quality of the submitted results is hard, malicious workers often take advantage of the verification difficulty and submit answers of low quality. Currently, most requesters rely on redundancy to identify the correct answers. However, redundancy is not a panacea. Massive redundancy is expensive, increasing significantly the cost of crowdsourced solutions. Therefore, we need techniques that will accurately estimate the quality of the workers, allowing for the rejection and blocking of the low-performing workers and spammers.However, existing techniques cannot separate the true (unrecoverable) error rate from the (recoverable) biases that some workers exhibit. This lack of separation leads to incorrect assessments of a worker's quality. We present algorithms that improve the existing state-of-the-art techniques, enabling the separation of bias and error. Our algorithm generates a scalar score representing the inherent quality of each worker. We illustrate how to incorporate cost-sensitive classification errors in the overall framework and how to seamlessly integrate unsupervised and supervised techniques for inferring the quality of the workers. We present experimental results demonstrating the performance of the proposed algorithm under a variety of settings.

The natural tendency for humans to make and break relationships is thought to facilitate the emergence of cooperation. In particular, allowing conditional cooperators to choose with whom they interact is believed to reinforce the rewards accruing to mutual cooperation while simultaneously excluding defectors. Here we report on a series of human subjects experiments in which groups of 24 participants played an iterated prisoner's dilemma game where, critically, they were also allowed to propose and delete links to players of their own choosing at some variable rate. Over a wide variety of parameter settings and initial conditions, we found that dynamic partner updating significantly increased the level of cooperation, the average payoffs to players, and the assortativity between cooperators. Even relatively slow update rates were sufficient to produce large effects, while subsequent increases to the update rate had progressively smaller, but still positive, effects. For standard prisoner's dilemma payoffs, we also found that assortativity resulted predominantly from cooperators avoiding defectors, not by severing ties with defecting partners, and that cooperation correspondingly suffered. Finally, by modifying the payoffs to satisfy two novel conditions, we found that cooperators did punish defectors by severing ties, leading to higher levels of cooperation that persisted for longer.network science | dynamic networks | web-based experiments W hy, and under what circumstances, presumptively selfish individuals cooperate is a question of longstanding interest to social science (1, 2) and one that has been studied extensively in laboratory experiments, often as some variant of a public goods game (also called a voluntary contribution mechanism) or a prisoner's dilemma (see refs. 2 and 3 for surveys). One broadly replicated result from this literature is that in unmodified, finitely repeated games cooperation levels typically start around 50-60% and steadily decline to around 10% in the final few rounds (2). The explanation for this decline is that when conditional cooperators, who are thought to constitute as much as 50% of the population (4), are forced to interact with a heterogeneous mix of other types, in particular free riders, the conditional cooperators tend to reduce their contributions over time (3).Previous work has shown that explicit enforcement mechanisms such as punishment (5) and reward (6) can alleviate the observed decline in cooperation in fixed groups. Here we investigate a related but distinct mechanism that exploits a wellknown feature of human social interactions-namely that they change over time, as individuals form new relationships or sever existing ones (7). By allowing participants engaged in a repeated game of cooperation to update their interaction partners dynamically, cooperation levels might be enhanced in two ways. First, conditional cooperators could implicitly punish defectors by denying them a partner, thereby encouraging potential defectors to cooperate. Second, conditional coope...

This paper addresses the repeated acquisition of labels for data items when the labeling is imperfect.We examine the improvement (or lack thereof) in data quality via repeated labeling, and focus especially on the improvement of training labels for supervised induction. With the outsourcing of small tasks becoming easier, for example via Amazon's Mechanical Turk, it often is possible to obtain less-than-expert labeling at low cost. With low-cost labeling, preparing the unlabeled part of the data can become considerably more expensive than labeling. We present repeated-labeling strategies of increasing complexity, and show several main results. (i) Repeated-labeling can improve label quality and model quality, but not always.(ii) When labels are noisy, repeated labeling can be preferable to single labeling even in the traditional setting where labels are not particularly cheap. (iii) As soon as the cost of processing the unlabeled data is not free, even the simple strategy of labeling everything multiple times can give considerable advantage.(iv) Repeatedly labeling a carefully chosen set of points is generally preferable, and we present a set of robust techniques that combine different notions of uncertainty to select data points for which quality should be improved. The bottom line: the results show clearly that when labeling is not perfect, selective acquisition of multiple labels is a strategy that data miners should have in their repertoire; for certain label-quality/cost regimes, the benefit is substantial.