One expensive step when defining crowdsourcing tasks is to define the examples and control questions for instructing the crowd workers. In this paper, we introduce a self-training strategy for crowdsourcing. The main idea is to use an automatic classifier, trained on weakly supervised data, to select examples associated with high confidence. These are used by our automatic agent to explain the task to crowd workers with a question answering approach. We compared our relation extraction system trained with data annotated (i) with distant supervision and (ii) by workers instructed with our approach. The analysis shows that our method relatively improves the relation extraction system by about 11% in F1.

In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: a classi er (i.e., the machine) selects the highest quality examples for training crowdsource workers (i.e., the humans). en, the la er annotate the lower quality examples such that the classi er can be re-trained with more accurate examples. is process can be iterated several times. We tested our approach on two di erent tasks, Relation Extraction and Community estion Answering, which are also in two di erent languages, English and Arabic, respectively. Our experimental results show a signi cant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. Additionally, our method can approach the performance of the state-of-the-art methods that use expensive Gold Standard for training workers. CCS CONCEPTS•Information systems →Retrieval models and ranking; Learning to rank; estion answering; •Computing methodologies →Learning paradigms; Active learning se ings;

We propose Multi-task learning (MTL) for time-continuous or dynamic emotion (valence and arousal) estimation in movie scenes. Since compiling annotated training data for dynamic emotion prediction is tedious, we employ crowdsourcing for the same. Even though the crowdworkers come from various demographics, we demonstrate that MTL can effectively discover (1) consistent patterns in their dynamic emotion perception, and (2) the low-level audio and video features that contribute to their valence, arousal (VA) elicitation. Finally, we show that MTL-based regression models, which simultaneously learn the relationship between low-level audio-visual features and high-level VA ratings from a collection of movie scenes, can predict VA ratings for time-contiguous snippets from each scene more effectively than scene-specific models.

English. Recent years have seen an impressive development and diffusion of web applications to food domains, e.g., Yelp, TripAdvisors. These mainly exploit text for searching and retrieving food facilities, e.g., restaurants, caffé, pizzerias. The main features of such applications are: the location and quality of the facilities, where quality is extrapolated by the users' reviews. More recent options also enable search based on restaurant categorization, e.g., Japanese, Italian, Mexican. In this work, we introduce Appetitoso 1 , an innovative approach for finding restaurants based on the dishes a user would like to taste rather than using the name of food facilities or their general categories.Italiano. Recentemente siè assistito ad un impressionante sviluppo e diffusione di applicazioni web per il dominio del cibo, e.g., Yelp, TripAdvisors. Queste sfruttano principalmente il testo per la ricerca e il recupero di punti di ristoro, e.g., ristoranti, bar, pizzerie. Le caratteristiche principali usate dalle applicazioni sono: la posizione e la qualitá delle strutture che servono il cibo, dove la qualitáé estrapolata dalle recensioni degli utenti. Opzioni piú recenti consentono anche la ricerca in base alla categoria del ristorante, e.g., Giapponese, Italiano, Messicano. Questo articolo introduce Appetitoso, un nuovo modo di trovare punti di ristoro sulla base dei piatti che il cliente vuole gustare invece che sul nome del ristorante o su categories generali.

In this paper, we propose methods to take into account the disagreement between crowd annotators as well as their skills for weighting instances in learning algorithms. The latter can thus better deal with noise in the annotation and produce higher accuracy. We created two passage reranking datasets: one with crowdsource platform, and the second with an expert who completely revised the crowd annotation. Our experiments show that our weighting approach reduces noise improving passage reranking up to 1.47% and 1.85% on MRR and P@1, respectively.