Completing self-administered patient experience surveys is challenging for many patients. We randomized adult patients receiving care from an urban safety net provider to complete the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Clinician and Group Survey 3.1 (CG-CAHPS 3.1), or an “EZ” survey created using plain language principles. We compared response rates, item missingness, item-scale correlations, and reliability of patient experience scores based on 264 completed surveys (64% female, 66% Hispanic, 33% high school education or less). The CG-CAHPS 3.1 survey response rate was higher (20% vs 16%), and failure to follow skip instructions was more common for the EZ survey. Internal consistency reliability for multi-item scales was similar, but provider-level reliability was higher for the EZ than for the CG-CAHPS 3.1 survey measures. Cognitive interviews with patients are needed to assess whether the wording of the EZ survey is responsible for the lower response rates and more skip pattern errors. Future studies are also required to provide additional information about the psychometric properties of the CG-CAHPS 3.1 and EZ surveys.