Pay Attention, Please

Abstract

Participant engagement in online studies is key to collecting reliable data, yet achieving it remains an often discussed challenge in the research community. One factor that might impact engagement is the formality of language used to communicate with participants throughout the study. Prior work has found that language formality can convey social cues and power hierarchies, affecting people's responses and actions. We explore how formality influences engagement, measured by attention, dropout, time spent on the study and participant performance, in an online study with 369 participants on Mechanical Turk (paid) and LabintheWild (volunteer). Formal language improves participant attention compared to using casual language in both paid and volunteer conditions, but does not affect dropout, time spent, or participant performance. We suggest using more formal language in studies containing complex tasks where fully reading instructions is especially important. We also highlight trade-offs that different recruitment incentives provide in online experimentation.

Constraints on German <i>diese</i> demonstratives: language formality and subject-avoidance

“…They found that the ethnic variety lead to stronger agreeability than the formal variety, and the informal and ethnic variations gave rise to higher relatedness with the speaker. At the perception level, language formality has also been shown to improve participants' attention during online experiments (August & Reinecke 2019). In another exploratory sociophonetic study, Winter & Grawunder (2012) show that language formality modulates a number of vocal expressions in Korean.…”

Know Your Audience: The benefits and pitfalls of generating plain language summaries beyond the "general" audience

“…Table 3 lists demographics and topic familiarity. Extrinsic motivations like payment can lead participants to maximize pay at the expense of data quality (e.g., by rushing through a study [9,106]). Studies 1 and 3 used Master Turkers, who have been shown to provide data quality equivalent to intrinsically motivated participants (e.g., participants motivated by supporting science) [106].…”

“…Adapting interfaces to different users has the potential to improve usability [13,87], aesthetic judgements [28,69], and trust [61,68]. Increasingly, language styles, such as community language norms [26], formality [9], and text complexity [11,95] have been the focus of adaptable user interfaces. Work has shown that language styles can impact behavior in online experiments [9], counseling conversations [4], online communities [26], and security interfaces [98].…”

“…Increasingly, language styles, such as community language norms [26], formality [9], and text complexity [11,95] have been the focus of adaptable user interfaces. Work has shown that language styles can impact behavior in online experiments [9], counseling conversations [4], online communities [26], and security interfaces [98]. This work has highlighted the benefits of adapting language to people with different backgrounds [11].…”

Design recommendations for augmenting creative tasks with computational priming

“…Nguyen et al showed that framing a task with positive affective language may reduce the frustration of participants and lead to an increase in work quality [48]. The framing of a task may also influence the attention of a worker [9]. Morris et al applied affective priming theory in crowdsourcing [46].…”