Agreement Between Self-reports and Photos to Assess e-Cigarette Device and Liquid Characteristics in Wave 1 of the Vaping and Patterns of e-Cigarette Use Research Study: Web-Based Longitudinal Cohort Study

Abstract

Background e-Cigarette device and liquid characteristics are highly customizable; these characteristics impact nicotine delivery and exposure to toxic constituents. It is critical to understand optimal methods for measuring these characteristics to accurately assess their impacts on user behavior and health. Objective To inform future survey development, we assessed the agreement between responses from survey participants (self-reports) and photos uploaded by participants and the quantity of usable data derived from each approach. Methods Adult regular e-cigarette users (≥5 days per week) aged ≥21 years (N=1209) were asked questions about and submitted photos of their most used e-cigarette device (1209/1209, 100%) and liquid (1132/1209, 93.63%). Device variables assessed included brand, model, reusability, refillability, display, and adjustable power. Liquid variables included brand, flavor, nicotine concentration, nicotine formulation, and bottle size. For each variable, percentage agreement was calculated where self-report and photo data were available. Krippendorff α and intraclass correlation coefficient (ICC) were calculated for categorical and continuous variables, respectively. Results were stratified by device (disposable, reusable with disposable pods or cartridges, and reusable with refillable pods, cartridges, or tanks) and liquid (customized and noncustomized) type. The sample size for each calculation ranged from 3.89% (47/1209; model of disposable devices) to 95.12% (1150/1209; device reusability). Results Percentage agreement between photos and self-reports was substantial to very high across device and liquid types for all variables except nicotine concentration. These results are consistent with Krippendorff α calculations, except where prevalence bias was suspected. ICC results for nicotine concentration and bottle size were lower than percentage agreement, likely because ICC accounts for the level of disagreement between values. Agreement varied by device and liquid type. For example, percentage agreement for device brand was higher among users of reusable devices (94%) than among users of disposable devices (75%). Low percentage agreement may result from poor participant knowledge of characteristics, user modifications of devices inconsistent with manufacturer-intended use, inaccurate or incomplete information on websites, or photo submissions that are not a participant’s most used device or liquid. The number of excluded values (eg, self-report was “don’t know” or no photo submitted) differed between self-reports and photos; for questions asked to participants, self-reports had more usable data than photos for all variables except device model and nicotine formulation. Conclusions Photos and self-reports yield data of similar accuracy for most variables assessed in this study: device brand, device model, reusability, adjustable power, display, refillability, liquid brand, flavor, and bottle size. Self-reports provided more data for all variables except device model and nicotine formulation. Using these approaches simultaneously may optimize data quantity and quality. Future research should examine how to assess nicotine concentration and variables not included in this study (eg, wattage and resistance) and the resource requirements of these approaches.

Device and liquid characteristics used with sweet, menthol/mint, and tobacco ENDS liquid flavors: The population-based VAPER study

“…Nicotine concentration, on the other hand, presents additional measurement challenges; however, the strategy utilized in this paper in which photo and self-report data are used in tandem may optimize data quality. ( Crespi et al, 2022 ).…”

Transitions in device and liquid characteristic groupings among US adults frequently using electronic nicotine delivery systems (ENDS) over three timepoints, 2020–2021

“…Additionally, post-stratification weights ranging from 0.64 to 1.56 were applied to improve representativeness. The data quality and quantity is optimized by employing a range of data integrity procedures and by using photo data of devices and liquids 13,29 . We examined the transition of ENDS device/liquid grouping rather than device type alone, which allows exploring key variability influencing nicotine delivery 14,30 .…”

Decisions of the FDA on premarket tobacco product applications: Changes in the number of unique devices and liquids used by US adults who frequently use electronic nicotine delivery systems, 2020–2023

“…The data reported here were captured directly from participant photos of their devices and liquids and from manufacturer, academic, retail, and review sites; therefore, these data are only as accurate as the photos and sites utilized for coding. However, previous research has suggested that self-reported and photo-coded device type and flavor have a high agreement, suggesting a fair level of accuracy with these data 36 . As mentioned in the results, the increased identification of products across waves may account for some of the increase in liquid products across waves; however, the increase in the proportion of liquids identified is more likely driven by the decreasing prevalence of disposable pod devices, for which photos often do not include the flavor or nicotine concentration of the product because this information is often not included on the pod itself.…”