The influence of automated praise on behavior and performance.

Abstract

Positive feedback has known benefits for improving task performance, but it is not clear why. On the one hand, positive feedback may direct attention to the task and one's motivations for having performed the task. On the other hand, positive feedback may direct attention to the task's value for achieving a future goal. This ambiguity presents a challenge for the design of automated feedback interventions. Specifically, it is unclear whether positive feedback will more effectively influence behavior when it praises the recipient for having performed an action, or when it highlights the action's value toward a goal. In the present study, we test these competing approaches in a large-scale field experiment (n = 1,766). Using a mobile app, we assigned college students to receive occasional notifications immediately upon submitting online assignments that either praised them for having submitted their coursework, or that highlighted the value of submitting coursework for academic success, or to a no-treatment control group. We find that only praise messages improved submission rates and course performance, suggesting that drawing attention to the feedback-eliciting task is necessary and sufficient for influencing behavior at scale.

Concentration toward the mode: Estimating changes in the shape of a distribution of student data

“…Additionally, considering that the difference in concentration can be difficult to conceptualize in practical terms, we can conduct the descriptive post-hoc analysis from Motz et al (2021). This analysis involves finding the score where the proportion of observations at or above this value is maximally different between groups.…”

“…Given that the mode was at ceiling, enthusiastic instructions had no room to increase participants' modal tendency to provide a response, but enthusiastic instructions did increase concentration toward the mode, causing participants who would otherwise leave items blank to be more likely to provide responses on delayed test items. And in Motz et al (2021), undergraduate students using a mobile app either received occasional positive feedback notifications for making on-time assignment submissions in the learning management system, or were assigned to a control condition that did not receive feedback. Most students submit all their assignments, yielding a skewed distribution with a mode at ceiling, but positive feedback increased the concentration of students' assignment submission rates toward the mode, causing students who might otherwise have missed assignment deadlines to make more on-time submissions.…”

“…There are many possible models, using a variety of distribution functions, that can be used to quantify concentration; selection of a model should be considered for its appropriateness to the structure of the data under analysis. Among these, the current article highlights a beta-binomial model (Wilcox, 1981), and this specific approach has been used in published research on learner behavior and performance (Motz et al, 2017(Motz et al, , 2021.…”

“…In our beta-binomial model, ω is the central tendency of a distribution of probabilities -draws from a binomial distribution -so in principle the prior for ω can be any value in the range [0,1]. If we wanted to use a vague prior, we could set this as a uniform distribution with equal density from 0 to 1 (as was the case in Motz et al, 2017Motz et al, , 2021. However, for our current analytical goals, there are at least two reasons to opt for a more informed prior.…”

“…Special thanks to Emily Fyfe for posting her data (which is remarkably well organized and documented) for secondary analysis, and for her constructive comments on an early version of this manuscript. Thanks also to Maria Ramirez and the SoLAR Methodology SIG for introducing me to QuantCrit, to Reviewer #2 who anonymously reviewed Motz et al (2021) and repetitively insisted that I should describe this approach in further detail. Thanks also to Tim Pleskac and the Mesolab for helpful pointers.…”