Crowdsourcing is used in academia and industry to solve tasks that are easy for humans but hard for computers, in natural language processing mostly to annotate data. The quality of annotations is affected by problems in the task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. To learn about the major problems, we provide a short but comprehensive survey based on two complementary studies: (1) a literature review where we collect and organize problems known from interviews with workers, and (2) an empirical data analysis where we use topic modeling to mine workers' complaints from a new English corpus of workers' forum discussions. While literature covers all process phases, problems in the task evaluation are prevalent, including unfair rejections, late payments, and unjustified blockings of workers. According to the data, however, poor task design in terms of malfunctioning environments, bad workload estimation, and privacy violations seems to bother the workers most. Our findings form the basis for future research on how to improve crowdsourcing processes. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. create task Platform (1) Task design (2) Task operation (3) Task evaluation assess results post task take on task solve task obtain results submit results accept or reject get paid or not questions, inquiries responses, feedback explanations, responses complaints Requester Workers Figure 1: The three phases of a crowdsourcing process. (1) Task design: The requester creates a task.(2) Task operation: Workers accept and solve the task, and submit the results.(3) Task evaluation: The requester accept or reject the results. Communication may happen during task operation and evaluation.approaches in this paper, bringing together knowledge from human computation with ideas from NLP: First, a literature review where we collect crowdsourcing problems discussed in related work, mostly obtained from interviews with and surveys among workers (Section 3). We consolidated the reported challenges from a wide range of studies in order to form a unified view on problems already known in literature. Second, an empirical data analysis where we use topic modeling to mine the workers' problems from what they complain about on a daily basis in online discussion forums (Section 4). Given its dominance, we focus on MTurk here, building a new corpus of 27,041 reviews from Turkopticon, a reputation portal where MTurk workers write reviews on requesters. In our analysis, we focus on the 8,610 negative reviews, since they are likely to contain complaints. 1 The hypothesis underlying our combined approach is that the literature gives a more comprehensive view of problems existing in crowdsourcing, whereas the data better reflects their significance. In light Rof this hypothesis, we study two research questions:RQ1. What problems do workers face in the different phases of crowdsour...