Reliable, valid and efficient evaluation of media messages

Abstract

Purpose In the field of public relations and communication management, message evaluation has been one of the starting points for evaluation and measurement research at least since the 1970s. Reliable and valid message evaluation has a central role in message effects research and campaign design in other disciplines as well as communication science. The purpose of this paper is to offer a message testing protocol to efficiently acquire valid and reliable message evaluation data. Design/methodology/approach A message testing protocol is described in terms of how to conceptualize and evaluate the content and format of messages, in terms of procedures for acquiring and testing messages and in terms of using efficient, reliable and valid measures of perceived message effectiveness (PME) and perceived argument strength (PAS). The evidence supporting the reliability and validity of PME and PAS measures is reviewed. Findings The message testing protocol developed and reported is an efficient, reliable and valid approach for testing large numbers of messages. Research limitations/implications Researchers’ ability to select candidate messages for subsequent deeper testing, for various types of communication campaigns, and for research in theory testing contexts is facilitated. Avoiding the limitations of using a single instance of a message to represent a category (also known as the case-category confound) is reduced. Practical implications Communication campaign designers are armed with tools to assess messages and campaign concepts quickly and efficiently, reducing pre-testing time and resources while identifying “best-in-show” examples and prototypes. Originality/value Message structures are conceptualized in terms of content and format features using theoretically driven constructs. Measures of PAS and PME are reviewed for their reliability, construct and predictive validity, finding that the measures are acceptable surrogates for actual effectiveness for a wide variety of messages and applications. Coupled with procedures that reduce confounding by randomly nesting messages within respondents and respondents to messages, the measures used and protocol deployed offer an efficient and utilitarian approach to message testing and modeling.

Peer crowd-based targeting in E-cigarette advertisements: a qualitative study to inform counter-marketing

“…This widely used tactic in community-based participatory research (e.g., [67,68]) could be leveraged for e-cigarette counter-marketing campaigns and interventions. Authenticity can also be improved through rigorous pre-testing with the target audience [69] to detect psychological reactance and negative emotions.…”

Breaking out: the turning point in learning using mobile technology

“…While reducing the technological complexity of some learning content may—in some cases—reduce the visual appeal of the educational content, the alternative is to fail to engage large parts of the public due to a lack of technical capacity. Although much effort goes into the “wow” factor when it comes to mobile content, for the public at large, keeping content delivery platforms as simple as possible best serves reaching the largest number of people possible ( Kim and Cappella, 2019 ).…”

The Effectiveness of Online Messages for Promoting Smoking Cessation Resources: Predicting Nationwide Campaign Effects From Neural Responses in the EX Campaign

“…However, given that individual variability exists in how both the neural and the online samples respond to each message, another question will be how many individuals are needed to obtain robust group-level metrics. Kim and Cappella ( 2019 ) suggest that collapsing ratings from about 25 raters per message provides stable ratings when norming health messages for constructs like perceived effectiveness or perceived argument strength. Prior neuroimaging studies aiming to use average brain responses from small groups to predict larger campaign success have used sample sizes between 18 and 47 (Knutson and Genevsky, 2018 ).…”

“…In the case of tobacco, for example, prevention efforts have only about 5% of the budget available to tobacco companies (Farrelly et al, 2003 ; CDC/CDCTobaccoFree, 2019 ). Despite important progress (Cho, 2011 ; Kim and Cappella, 2019 ; Sutton et al, 2019 ), the creation and selection of campaign messages remain a mix of art and science (Noar, 2011 ; Rice and Atkin, 2012 ). A deeper understanding of the factors associated with message success is critical for designing more efficient and effective health campaigns.…”