Community partners’ responses to items assessing stakeholder engagement: Cognitive response testing in measure development

Abstract

Background Despite recognition of the importance of stakeholder input into research, there is a lack of validated measures to assess how well constituencies are engaged and their input integrated into research design. Measurement theory suggests that a community engagement measure should use clear and simple language and capture important components of underlying constructs, resulting in a valid measure that is accessible to a broad audience. Objective The primary objective of this study was to evaluate how community members understood and responded to a measure of community engagement developed to be reliable, valid, easily administered, and broadly usable. Method Cognitive response interviews were completed, during which participants described their reactions to items and how they processed them. Participants were asked to interpret item meaning, paraphrase items, and identify difficult or problematic terms and phrases, as well as provide any concerns with response options while responding to 16 of 32 survey items. Results The results of the cognitive response interviews of participants (N = 16) suggest concerns about plain language and literacy, clarity of question focus, and the lack of context clues to facilitate processing in response to items querying research experience. Minimal concerns were related to response options. Participants suggested changes in words and terms, as well as item structure. Conclusion Qualitative research can improve the validity and accessibility of measures that assess stakeholder experience of community-engaged research. The findings suggest wording and sentence structure changes that improve ability to assess implementation of community engagement and its impact on research outcomes.

Development and Validation of a Brief Version of the Research Engagement Survey Tool

“…We performed a mixed-methods study (using both quantitative and qualitative research approaches) that consisted of a modified Delphi process (conducted between July 2017 and August 2018) [ 1 , 2 ], cognitive interviews (conducted in October 2018) [ 4 ], and 4 longitudinal web-based surveys (conducted between July 2017 and September 2019). All participants were currently or previously involved in community-engaged research [ 1 , 2 , 3 , 4 ]. We used triangulation analysis on data from the Delphi process, cognitive interviews, and web surveys to guide the development of a brief version of the REST.…”

“…In an effort to identify potential modifications to be made to the measure, we aimed to use the cognitive interviews to determine how participants responded to and understood the items in the measure. The cognitive interview process has been described in detail elsewhere [ 3 , 4 ].…”

“…On the basis of the results described above, we created the condensed REST. Items that were non-loading or were cross-loading in the factor analysis (supplemental Table S2 4. Using these criteria, all items from EP1 would have been cut.…”

“…In our previous work, we developed and validated the comprehensive (32-item) Research Engagement Survey Tool (REST) to examine the level of non-academic stakeholder engagement among research partners [ 1 , 2 , 3 , 4 ]. The REST is based on eight engagement principles (EPs) from the research literature on community engagement, community–academic partnerships, community-based participatory research, and patient-centered outcomes research [ 1 ].…”

A Study Examining the Usefulness of a New Measure of Research Engagement

“…The REST is a valid and reliable quantitative survey tool to assess research engagement of non-academic stakeholders in the research process [ 2 , 9 ]. Project teams can use this information to examine engagement strategies and the level of engagement of a project.…”