“…Another consideration for our remote, novice programming proposal is whether this work can be crowdsourced effectively. The success of crowdsourcing in other tasks has been shown extensively in prior work, including for spoken dialog generation for conversational systems (Lasecki et al, 2013;Mitchell et al, 2014;Leite et al, 2016;Yu et al, 2016;Guo et al, 2017;Kennedy et al, 2017;Huang et al, 2018;Jonell et al, 2019) and interaction data and non-verbal behavior (Orkin and Roy, 2007;Orkin and Roy, 2009;Chernova et al, 2010;Rossen and Lok, 2012;Breazeal et al, 2013;Sung et al, 2016). In previous research that is, more relevant to ours, Lee and Ko (2011) crowdsourced non-expert programmers for an online study and found that personified feedback of a robot blaming itself for errors increased the non-programmers' motivation to program.…”