Survey Protocols, Response Rates, and Representation of Underserved Patients

Abstract

ImportanceSurveys often underrepresent certain patients, such as underserved patients. Methods that improve their response rates (RRs) would help patient surveys better represent their experiences and assess equity and equity-targeted quality improvement efforts.ObjectiveTo estimate the effect of adding an initial web mode to existing Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey protocols and extending the fielding period on RR and representativeness of underserved patient groups.Design, Setting, and ParticipantsThis randomized clinical trial included 36 001 patients discharged from 46 US hospitals from May through December 2021. Data analysis was performed from May 2022 to September 2023.ExposuresPatients were randomized to 1 of 6 survey protocols: 3 standard HCAHPS protocols (mail only, phone only, mail-phone) plus 3 web-enhanced protocols (web-mail, web-phone, web-mail-phone).Main Outcomes and MeasuresRR and number of respondents per 100 survey attempts (yield) were calculated and compared for each of the 6 survey protocols, overall, and by patient age, service line, sex, and race and ethnicity.ResultsA total of 34 335 patients (median age range, 55-59 years; 59.3% female individuals and 40.7% male individuals) were eligible and included in the study. Of the respondents, 6.9% were Asian American or Native Hawaiian or Other Pacific Islander, 0.7% were American Indian or Alaska Native, 11.5% were Black, 17.4% were Hispanic, 61.0% were White, and 2.6% were multiracial. Of the 6 protocols, RRs were highest in web-mail-phone (36.5%), intermediate for the 3 two-mode survey protocols (mail-phone, web-mail, web-phone, 30.3%-31.1%), and lowest for the 2 single-mode protocols (mail only, phone only, 22.1%-24.3%). Web-mail-phone resulted in the highest yield for 3 racial and ethnic groups (Black, Hispanic, and White patients) and second highest for another (multiracial patients). Otherwise, the highest or second highest yield was almost always a 2-mode protocol. Mail only was the lowest-yield protocol for Black, Hispanic, and multiracial patients and phone-only was the lowest-yield protocol for White patients; these 2 protocols tied for lowest-yield for Asian American or Native Hawaiian or Other Pacific Islander patients. Gains from multimode approaches were often 2 to 3 times as large for Asian American or Native Hawaiian or Other Pacific Islander, Black, Hispanic, and multiracial patients as for White patients. Web-mail-phone had the highest RR for 6 of 8 age groups and 4 of 5 combinations of service line and sex.Conclusions and RelevanceIn this randomized clinical trial, web-first multimode survey protocols significantly improved the RR and representativeness of patient surveys. The best-performing protocol based on RR and representativeness was web-mail-phone. Web-phone performed well for young and diverse patient populations, and web-mail for older and less diverse patient populations. The US Centers for Medicare & Medicaid Services will allow hospitals to use the web-mail, web-phone, and web-mail-phone protocols for HCAHPS administration beginning in 2025.

Patient-Reported Outcome Measures Within a National Multispecialty Surgical Quality Improvement Program

“…Ninety percent of hospitals achieved the target collection rate of 30% or greater, and 9 exceeded 50%, suggesting that further substantive increases are possible with continued process improvements. Based on existing experience with national health surveys in the US, collection rates range between 30% and 50% . Although lower than survey research standards, high-stakes decisions are nonetheless made with these data.…”

“…Although lower than survey research standards, high-stakes decisions are nonetheless made with these data. For instance, the Consumer Assessment of Healthcare Providers and Systems Hospital Survey achieved a median 31% collection rate in 2021 to 2022 using mail and telephone modes, and changes to increase responses are imminent . Further, the Centers for Medicare & Medicaid Services now mandates the collection of PROMs to evaluate the care quality of total joint replacement with significant financial implications—the target collection rate to satisfy the performance measure is 50% or greater.…”

“…The primary metrics to evaluate implementation were hospital participation (adoption) and the collection rate (reach), calculated as 1 or more PROMs submitted per patient over either the project duration or per quarter by hospital . Because the collection rates of current national programs in the US to evaluate health care delivery using standardized questionnaires range from approximately 30% to 50%, the target collection rate for this unprecedented effort was reasonably set at 30% or greater.…”

“…To demonstrate adoption, our primary goal was to accrue more than 30 hospitals in the first 3 years by following the Institute for Healthcare Improvement (IHI) Framework for Spread and the Consolidated Framework for Implementation Research (CFIR) . Consistent with the collection rates of national patient-reported survey-based programs, the target collection rate was set at 30% or greater to demonstrate reach. We report the implementation strategies conducted to increase collection rates, explore hospital factors contributing to increased collection rates, and characterize nonresponse bias.…”

Error in Figure 2

“…In the Original Investigation titled “Survey Protocols, Response Rates, and Representation of Underserved Patients: A Randomized Clinical Trial” 1 published on January 19, 2024, there were errors in Figure 2. Panel B was replaced because it did not previously reflect the data meant to be displayed.…”