Message Pretesting Using Perceived Persuasiveness Measures: Reconsidering the Correlational Evidence

Abstract

Perceived message effectiveness (PME) measures have been used in persuasive message pretesting to diagnose differences between messages in actual message effectiveness (AME). This practice has been underwritten by pointing to positive correlations between individuals' PME and AME scores. This essay argues that such correlations do notand cannotshow that messages' relative standing on PME matches their relative standing on AME, and thus correlations between individuals' PME and AME scores are irrelevant to the question of whether PME assessments accurately identify relatively more effective messages. Only correlations between messages' PME and AME scores can indicate whether PME measures provide successful prediction of messages' relative AME standing. But positive correlations between individuals' PME and AME scores do suggest a possible repurposing of PME measures in formative research.

A Tailored Motivational Messages Library for a Mobile Health Sleep Behavior Change Support System to Promote Continuous Positive Airway Pressure Use Among Patients With Obstructive Sleep Apnea: Development, Content Validation, and Testing

“…In phase 3, the messages were further tested using a convenience sample of the target audience for perceived persuasiveness. Pretesting potential messages to identify the ones that are more likely to be relatively effective is an important step to assist the message designer or researcher in the process of selecting messages to include in the intervention [ 61 ].…”

Perceived Message Effectiveness and Campaign-Targeted Beliefs: Evidence of Reciprocal Effects in Youth Tobacco Prevention

“…This focus on change, together with a time lag and the separation of between-and within-individual processes, enables cleaner causal inference. It also helps alleviate, to some extent, a major concern about PE measurement, that is, the confound between message differences (external shocks that produce within-individual changes in PE) and individual differences (between-individual variation in PE ratings) (for discussion of this issue, see Noar, Bell et al, 2018;O'Keefe, 2020;Yzer et al, 2015;Zhao et al, 2011).…”

“…1. The current evidence on PE has also been challenged from another angle (O'Keefe, 2020). In a recent commentary, O'Keefe (2020) argues that individual-level relationship between PE and AE is irrelevant to the question of whether PE can identify relatively effective or ineffective messages -only message-level correlation between PE and AE can serve that purpose.…”

“…Perceived message effectiveness (PE) has been widely used in campaign research, both as a tool to assess message potential in formative research (Dillard, Weber et al, 2007;Fishbein et al, 2002), and as a surveillance device to monitor audience receptivity during campaign implementation (Duke et al, 2015;Rath et al, 2019). The last few years have seen a surge of interest in this concept and its application, leading to increased awareness of some important gaps in the literature (Noar, Bell et al, 2018;O'Keefe, 2018aO'Keefe, , 2020Yzer et al, 2015). At the core of the ongoing discussion is the relationship between PE and the attitudinal and behavioral outcomes that campaigns are ultimately interested in (termed actual effectiveness [AE]).…”

“…At the core of the ongoing discussion is the relationship between PE and the attitudinal and behavioral outcomes that campaigns are ultimately interested in (termed actual effectiveness [AE]). While previous research suggested a robust relationship between PE and AE (Dillard, Shen et al, 2007;Dillard, Weber et al, 2007), recent reexamination of the literature has called this conclusion in question (O'Keefe, 2018a(O'Keefe, , 2020. Concurrently, there is also concern that many PE measures have been developed and put to use without rigorous validation within the proper campaign context (Noar, Bell et al, 2018).…”