This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
e-Cigarette device and liquid characteristics are highly customizable; these characteristics impact nicotine delivery and exposure to toxic constituents. It is critical to understand optimal methods for measuring these characteristics to accurately assess their impacts on user behavior and health.
To inform future survey development, we assessed the agreement between responses from survey participants (self-reports) and photos uploaded by participants and the quantity of usable data derived from each approach.
Adult regular e-cigarette users (≥5 days per week) aged ≥21 years (N=1209) were asked questions about and submitted photos of their most used e-cigarette device (1209/1209, 100%) and liquid (1132/1209, 93.63%). Device variables assessed included brand, model, reusability, refillability, display, and adjustable power. Liquid variables included brand, flavor, nicotine concentration, nicotine formulation, and bottle size. For each variable, percentage agreement was calculated where self-report and photo data were available. Krippendorff
Percentage agreement between photos and self-reports was substantial to very high across device and liquid types for all variables except nicotine concentration. These results are consistent with Krippendorff
Photos and self-reports yield data of similar accuracy for most variables assessed in this study: device brand, device model, reusability, adjustable power, display, refillability, liquid brand, flavor, and bottle size. Self-reports provided more data for all variables except device model and nicotine formulation. Using these approaches simultaneously may optimize data quantity and quality. Future research should examine how to assess nicotine concentration and variables not included in this study (eg, wattage and resistance) and the resource requirements of these approaches.
e-Cigarettes are devices that heat a liquid to produce an aerosol for inhalation. The device itself typically consists of a battery, an electrical heater (eg, an atomizer or coil), a container (eg, cartridge, pod, or tank) to hold liquid, and a mouthpiece for inhalation [
Various constituents that may be associated with negative health outcomes have been found in e-cigarette liquids and aerosols, including aldehydes, carbonyls, and heavy metals among others [
Self-report data in health research can be subject to several biases (eg, social desirability, recall period, sampling approach, or selective recall) that affect the validity and reliability of the data [
Data were from wave 1 (May 2020 to October 2020) of the Vaping and Patterns of E-cigarette Use Research study, which is a US-based longitudinal cohort study following regular vapers (≥5 days per week) aged ≥21 years through a web-based survey about e-cigarette use patterns and behaviors. Participants of wave 1 were recruited via web-based advertisements on Facebook, Instagram, and Craigslist and flyers and business cards distributed by vape shops. Advertisements were posted in 125 US cities selected for their potential to yield respondents who use e-cigarettes (ie, relatively high population).
After clicking an advertisement, participants were directed to a web-based survey hosted by Virginia Commonwealth University’s REDCap (Research Electronic Data Capture; Vanderbilt University), a Health Insurance Portability and Accountability Act–compliant secure web application for building and managing web-based surveys and databases. Before answering any questions, participants reviewed a consent form and certificate of confidentiality and were asked if they would like to continue with the survey. Then, they provided their contact information (ie, full name, phone number, email address, mailing address, and date of birth).
In addition to answering survey questions about their use patterns and behaviors, participants were asked to submit photos of their most used e-cigarette device and their most used liquid for that device. Upon completion, participants received US $10 Amazon gift codes via mail. To exclude potential bot activity, survey responses were reviewed for non-English or Spanish alphabet (0/2813, 0%), data suggesting inattention or very low knowledge of e-cigarettes or liquids (eg, indicating that they are aged 25 years and began vaping at the age of 50 years or self-reporting JUUL as a disposable device; 38/2813, 1.35%), invalid mailing addresses (13/2813, 0.46%), answering the minimum number of questions (this suggests participant may have previous knowledge of survey skip logic; 0/2813, 0%), completing the survey in <5 minutes (24/2813, 0.85%), missing attention check questions (64/2813, 2.28%), straight lining (0/2813, 0%), failed or incomplete identity authentication (eg, did not provide additional information such as utility bill to confirm identity; 765/2813, 27.2%), or multiple survey attempts (162/2813, 5.76%). Participants (464/2813, 16.5%) were also excluded if their device photos were from the internet (as determined by a Google search describing the image or a Google Reverse Image Search), submitted multiple times, not of an e-cigarette device, or taken in a store. JUUL, Vaporesso, and Voopoo were included as examples in the question prompt for device brand, and records suspected to be bots or professional survey takers frequently reported these 3 brands. Therefore, we decided to consider participants who self-reported one of these 3 brands and submitted a photo of a device that did not match the reported brand to be invalid. In all, 0.11% (3/2813) of the records were not reviewed for various reasons (eg, completed survey after the survey closed) and were excluded from the final data set.
Upon completion of the data collection wave, participants were also excluded if they were found to be highly suspicious in a review of suspicious records (71/2813, 2.52%). Records were comprehensively reviewed if they had incentives returned to the sender or contained data issues such as reporting devices intended for cannabis or oils or beginning to use e-cigarettes before they were commercially available in the United States. Records were considered highly suspicious and were excluded if they contained 1 significant data issue (eg, self-reported liquid contains cannabis derivatives) or a combination of multiple data issues (eg, mismatched self-report and photo device or liquid, patterned responses, or self-reported device or liquid characteristics that do not match self-reported device or liquid). A total of 7875 participants completed the screener, of which 4289 (54.46%) participants were eligible. Of the 4289 participants, 2813 (65.59%) participants completed the survey. Of the 2813 participants, 1604 (57.02%) were excluded after implementing the strategies listed above to avoid bots and professional survey takers, resulting in a sample size of 1209 (42.98%). Researchers (EC and JH) used the Google search engine to search text and markings in submitted photos of devices and liquids to identify the device brand and model and liquid brand and flavor by visually matching the submitted photo with Google search results. Then, the device brand and model and liquid brand and flavor were searched on Google to identify, in the following order of priority, manufacturer, academic (ie, journal articles), retail, and review sites for the given device and liquid and record information about key characteristics of the devices and liquids. One site was sufficient for confirming the information for a characteristic; however, up to 3 sites were searched for each characteristic before categorizing the information as missing. If discrepancies were found between sites for a characteristic, information from the site with highest priority was used. If the device brand and model or liquid brand and flavor could not be identified or the information for a particular characteristic could not be found in the photo or on the web, this information was considered missing. An initial round of reliability testing was conducted for this process to ensure high reliability (≥90%) between researchers.
All participants provided informed consent. The institutional review board at the Virginia Commonwealth University (no HM20015004) approved the study protocol. The Johns Hopkins Bloomberg School of Public Health Institutional Review Board (no 9277) approved reliance on the Virginia Commonwealth University Institutional Review Board.
Self-reported device brand and model were assessed using the following question: “What is the brand AND model of the device (eg, JUUL, Vaporesso Luxe, Voopoo Drag 2, etc.)?” As participants who self-reported JUUL, Vaporesso, or Voopoo were excluded if their self-reported and photo brand mismatched, these 3 self-reported brands were excluded from all device brand calculations (270/1209, 22.33%). Device brand and model from photos were determined by searching the identifying text or markings in the submitted photo on Google and finding a visual match on the web. For all calculations, device brand and model were assessed individually rather than as a combined variable.
Self-reported reusability of the device was assessed using the following question: “Is the device (1) reusable (ie, you recharge the device when the battery life is low or at 0%) or (2) disposable (ie, you discard entire device when the battery life is low or at 0%)?” For photos, websites obtained from the Google search conducted for each device were reviewed for mentions of the device being disposable or reusable. Then, the records were coded based on the information on the website.
Self-reported refillability of the device was assessed using the following question: “When the device runs out of e-liquid, do you TYPICALLY (1) discard the empty cartridge or pod and replace with a new and unused cartridge or pod prefilled with e-liquid or (2) refill the empty tank/cartridge/pod with e-liquid from a larger container(s) of e-liquid?” Given that disposable devices cannot be refillable, this question was asked only to participants who indicated that their device was reusable. The information on whether the device was refillable or nonrefillable was obtained from photos by searching the device brand and model on Google and extracting details from websites.
Self-reported presence of a visual representation of adjustable settings on the device was assessed using the following question: “Does the device have a VISUAL DISPLAY that allows you to see the wattage or other vape settings?” Response options included “yes” and “no.” This question was asked only to participants who indicated that their device was reusable. For photos, a visual representation of adjustable settings was defined as a display (eg, a screen, small light, or dial) that shows information about settings that can be adjusted by the user; this information was confirmed from websites for the given device.
Self-reported ability to adjust device power was assessed using the following question: “Does the device have SETTINGS that allow you to modify power or vapor volume?” Response options included “yes” and “no.” This question was asked only to participants who indicated that their device was reusable. For photos, a device was considered to have adjustable power if websites for the given device indicated that the user can customize the wattage or voltage using a dial or button (not by changing the coil or other internal parts).
Liquid type was assessed using participants’ responses to the following question: “Is your most used e-liquid a (1) customized flavor blend-mixed yourself, (2) customized flavor blend-mixed for you by someone else (3) non-customized flavor?” This question was asked only to users of reusable devices with refillable pods, cartridges, or tanks; users of disposable devices or reusable devices with disposable pods or cartridges were presumed to have noncustomized liquids. Agreement between photos and self-reports on liquid type was not assessed; rather, liquid type was used to stratify results by those using customized and noncustomized liquids.
Self-reported liquid brand was assessed using the following question: “Do you know the brand of the [cartridge or pod (eg, JUUL, blu, VUSE, etc.]/e-liquid container (eg, Naked 100, Beard Vape, Milkman, etc.)]? [If yes] please specify the brand.” As disposable devices do not have separate liquids, this question was not asked to users of disposable devices. Participants who indicated that they use a customized flavor blend were also excluded from this variable as they were not asked about the brand of their liquid. Photos of the liquids were assessed for brand by searching any identifying text or markings in the photo on Google and reviewing websites for a visual match.
Self-reported liquid flavor was assessed using the following question: “What is the flavor of the [device’s (for disposable devices)] e-liquid?” Response options included (1) tobacco; (2) tobacco menthol or menthol; (3) mint; (4) a flavor such as fruit, candy, alcohol, coffee, vanilla, or other food/drink; and (5) no flavor. Liquid flavors from photos were identified using flavor descriptions on the website for the given liquid, and then categorized using the e-cigarette liquid flavor wheel developed by Krusemann et al [
Self-reported nicotine concentration of the liquid was assessed using the following question: “Do you know how much nicotine is in the [device’s] e-liquid/flavor blend? [If yes] please specify.” Respondents indicated whether they reported the concentration as mg/mL or a percentage. When participants reported nicotine concentrations as a percentage, the reported value was multiplied by 10 to obtain the mg/mL; exceptions include JUUL, which is reported to be 59 mg/mL for their 5% pods and 35 mg/mL for their 3% pods [
Self-reported nicotine formulation of the liquid was assessed using the following question: “Does the [device’s] e-liquid/flavor blend contain nicotine salts?” Response options included “yes” and “no.” For photos, nicotine formulation was coded as either nicotine salt or free-base nicotine. As websites frequently report when a liquid is a salt but fail to report when a liquid is free-base and 27 mg/mL was the highest confirmed free-base nicotine liquid concentration in our sample, liquids with nicotine concentration ≤27 mg/mL, for which formulation could not be found on any website, were considered to be free-base. Liquids (2/1209, 0.17%) with nicotine concentrations >27 mg/mL, with formulation not found on any website (Vuse Solo Chai and Glas pods) were investigated further and considered to be salts because other liquids by the same brand are exclusively salts, YouTube reviewers indicated the liquid is likely a salt, or other similar devices frequently use salts.
Self-reported liquid bottle size was assessed using the following question: “Do you know the bottle size (in milliliters) of your most used e-liquid/flavor blend that you last purchased? [If yes] please specify the bottle size (in mL).” This question was asked only to participants who indicated their device was reusable and refillable. For photos, bottle size (in mL) was extracted from photos of the liquid bottles or from the manufacturer, academic, retail, or review sites. If the bottle size was not mentioned on the bottle and multiple sizes were available on the web for the given liquid, bottle size was considered missing for this liquid.
Percentage agreement, calculated as the number of records for which self-report and photo data were concordant divided by the total number of records with available self-report and photo data, was calculated for the following variables: device brand, device model, device reusability, device refillability, device display, adjustable power, liquid brand, liquid flavor, nicotine concentration, nicotine formulation, and liquid bottle size. As percentage agreement does not account for agreement expected by chance, Krippendorff
Percentage agreement, Krippendorff
Among our sample, 34.33% (415/1209) were aged 21-29 years, 36.15% (437/1209) were aged 30-39 years, 19.35% (234/1209) were aged 40-49 years, and the remaining 10.17% (123/1209) were aged ≥50 years (
Sociodemographic characteristics of survey participants (N=1209).
Sociodemographic characteristic | Participants, n (%) | ||
|
|||
|
Midwest | 216 (17.87) | |
|
Northeast | 130 (10.75) | |
|
South | 491 (40.61) | |
|
West | 372 (30.77) | |
|
|||
|
21-24 | 182 (15.05) | |
|
25-29 | 233 (19.27) | |
|
30-34 | 248 (20.51) | |
|
35-39 | 189 (15.63) | |
|
40-44 | 146 (12.08) | |
|
45-49 | 88 (7.28) | |
|
≥50 | 123 (10.17) | |
|
|||
|
Male | 545 (45.08) | |
|
Female | 646 (53.43) | |
|
Othera | 10 (0.83) | |
|
Prefer not to answer | 8 (0.66) | |
|
|||
|
American Indian or Alaska Native only | 9 (0.74) | |
|
Asian or Asian American only | 26 (2.15) | |
|
Black or African American only | 45 (3.72) | |
|
Native Hawaiian or Pacific Islander only | 4 (0.33) | |
|
White only | 920 (76.09) | |
|
Other | 42 (3.47) | |
|
Multiracial | 134 (11.08) | |
|
Prefer not to answer | 29 (2.39) | |
|
|||
|
Hispanic, Latino, or of Spanish origin | 129 (10.67) | |
|
Non-Hispanic, Latino, or of Spanish origin | 1061 (87.76) | |
|
Prefer not to answer | 19 (1.57) | |
|
|||
|
0-39,999 | 583 (48.22) | |
|
40,000-59,999 | 287 (23.74) | |
|
60,000-99,999 | 209 (17.29) | |
|
≥100,000 | 101 (8.35) | |
|
Prefer not to answer | 29 (2.39) | |
|
|||
|
5 | 74 (6.12) | |
|
6 | 26 (2.15) | |
|
7 | 1109 (91.73) | |
|
|||
|
Nonsmokerb | 711 (58.81) | |
|
Smokerb | 498 (41.19) |
aOther gender includes transgender individuals, nonbinary individuals, and so on.
bParticipants were considered non-smokers if they had not smoked in the past 30 days. Participants were considered smokers if they smoked cigarettes ≥1 time in the past 30 days.
The number of values excluded ranged from 4.88% (59/1209; device reusability) to 56.49% (683/1209; bottle size;
Excluded values for photos and self-report data by variable (N=1209).
Variables | Photo only, n (%) | Self-report only, n (%) | Both, n (%) | Total, n (%) | ||||||
|
|
Excluded for other reasonsa | Excluded because question was not asked to participant | Excluded for other reasonsa | Excluded because question was not asked to participant |
|
||||
Device brand | 42 (3.47) | 289b (23.90) | 0 (0) | 0 (0) | 0 (0) | 331 (27.38) | ||||
Device model | 29 (2.40) | 363 (30.02) | 0 (0) | 211 (17.45) | 0 (0) | 603 (49.88) | ||||
Device reusability | 59 (4.88) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 59 (4.88) | ||||
Device refillability | 51 (4.22) | 0 (0) | 111 (9.18) | 0 (0) | 8 (0.66) | 170 (14.06) | ||||
Device display | 51 (4.22) | 0 (0) | 111 (9.18) | 0 (0) | 8 (0.66) | 170 (14.06) | ||||
Adjustable power | 51 (4.22) | 0 (0) | 111 (9.18) | 0 (0) | 8 (0.66) | 170 (14.06) | ||||
Liquid brand | 70 (5.79) | 52 (4.3) | 307 (25.39) | 26 (2.15) | 106 (8.77) | 561 (46.40) | ||||
Liquid flavor | 317 (26.22) | 37 (3.06) | 0 (0) | 21 (1.74) | 0 (0) | 375 (31.02) | ||||
Nicotine concentration | 321 (26.55) | 50 (4.14) | 0 (0) | 63 (5.21) | 0 (0) | 434 (35.90) | ||||
Nicotine formulation | 224 (18.53) | 240 (19.85) | 0 (0) | 80 (6.62) | 0 (0) | 544 (45.00) | ||||
Liquid bottle size | 167 (13.81) | 72 (5.96) | 352 (29.11) | 37 (3.06) | 55 (4.55) | 683 (56.49) |
aOther reasons include if the participant self-reported “I don’t know” or the self-reported response was not able to be cleaned owing to lack of clarity (eg, liquid brand reported as “local vape shop”).
bIncludes records that were excluded because the brand was JUUL, Vaporesso, or Voopoo (n=270).
Sample size for calculations by liquid and device type.
Variables | Device types, n (%) | Liquid types, n (%) | |||||
|
Overall (n=1209) | DDa (n=119) | RDDb (n=288) | RDRc (n=802) | Noncustomized (n=915) | Customized (n=294) | |
Device brand | 878 (72.62) | 108 (90.8) | 141 (48.9) | 629 (78.4) | 645 (70.5) | 233 (79.3) | |
Device model | 606 (50.12) | 47 (39.5) | 57 (19.8) | 502 (62.6) | 447 (48.9) | 159 (54.1) | |
Device reusability | 1150 (95.12) | 111 (93.3) | 280 (97.2) | 759 (94.6) | 878 (95.9) | 272 (92.5) | |
Device refillability | 1039 (85.94) | N/Ad | 280 (97.2) | 759 (94.6) | 767 (83.8) | 272 (92.5) | |
Device display | 1039 (85.94) | N/A | 280 (97.2) | 759 (94.6) | 767 (83.8) | 272 (92.5) | |
Adjustable power | 1039 (85.94) | N/A | 280 (97.2) | 759 (94.6) | 767 (83.8) | 272 (92.5) | |
Liquid brand | 648 (53.59) | N/A | 251 (87.2) | 397 (49.5) | 648 (70.8) | N/A | |
Liquid flavor | 834 (68.98) | 92 (77.3) | 166 (57.6) | 576 (71.8) | 687 (75.1) | 147 (50) | |
Nicotine concentration | 775 (64.10) | 85 (71.4) | 137 (47.6) | 553 (68.9) | 602 (65.8) | 173 (58.8) | |
Nicotine formulation | 665 (55) | 61 (51.3) | 106 (36.8) | 498 (62.1) | 519 (56.7) | 146 (49.7) | |
Liquid bottle size | 526 (43.51) | N/A | N/A | 526 (65.6) | 382 (41.7) | 145 (49.3) |
aDD: disposable devices.
bRDD: reusable devices with disposable pods or cartridges.
cRDR: reusable devices with refillable pods, cartridges, or tanks.
dN/A: not applicable.
Percentage agreement was high (≥80%) between photos and self-reports for device reusability, adjustable power, device display, device refillability, and liquid brand (
These results were largely supported by Krippendorff
Results of percentage agreement calculations by liquid and device type.
Variables | Device types (%) | Liquid types (%) | |||||
|
Overall | DDa | RDDb | RDRc | Noncustomized | Customized | |
Device brand | 92 | 75 | 97.9 | 93.6 | 91.3 | 94 | |
Device model | 72.6 | 74.5 | 91.2 | 70.3 | 72.5 | 73 | |
Device reusability | 98.8 | 91 | 99.3 | 99.7 | 98.4 | 100 | |
Device refillability | 96.7 | N/Ad | 96.8 | 96.7 | 96.3 | 97.8 | |
Device display | 92.9 | N/A | 92.9 | 92.9 | 92.8 | 93 | |
Adjustable power | 93.9 | N/A | 97.1 | 92.8 | 95 | 90.8 | |
Liquid brand | 86.7 | N/A | 96 | 80.9 | 86.7 | N/A | |
Liquid flavor | 89.9 | 91.3 | 79.5 | 92.7 | 89.4 | 92.6 | |
Nicotine concentration | 66.3 | 65.9 | 56.2 | 69 | 68.1 | 60.1 | |
Nicotine formulation | 86.2 | 73.8 | 58.5 | 93.6 | 85 | 90.4 | |
Liquid bottle size | 74.3 | N/A | N/A | 74.3 | 78 | 64.6 |
aDD: disposable devices.
bRDD: reusable devices with disposable pods or cartridges.
cRDR: reusable devices with refillable pods, cartridges, or tanks.
dN/A: not applicable.
Results of Krippendorff α calculations by liquid and device type.
Variables | Device types (Krippendorff |
Liquid types (Krippendorff |
|||||
|
Overall | DDa | RDDb | RDRc | Noncustomized | Customized | |
Device reusability | .93 | −0.04 | 0 | 0 | .93 | 1 | |
Device refillability | .92 | N/Ad | −0.01 | −0.02 | .92 | −0.01 | |
Device display | .86 | N/A | .13 | .84 | .85 | .83 | |
Adjustable power | .88 | N/A | .32 | .82 | .90 | .74 | |
Liquid flavor | .78 | .61 | .66 | .69 | .79 | .45 | |
Nicotine formulation | .72 | −0.14 | −0.18 | .86 | .70 | .76 |
aDD: disposable devices.
bRDD: reusable devices with disposable pods or cartridges.
cRDR: reusable devices with refillable pods, cartridges, or tanks.
dN/A: not applicable.
Results of intraclass correlation coefficient calculations by liquid and device type.
Variables | Device types, estimate (95% CI) | Liquid types, estimate (95% CI) | |||||
|
Overall | DDa | RDDb | RDRc | Noncustomized | Customized | |
Nicotine concentration | 0.21 (0.14 to 0.27) | 0.01 (−0.14 to 0.18) | 0.04 (−0.12 to 0.20) | 0.18 (0.10 to 0.26) | 0.20 (0.13 to 0.28) | 0.16 (0.02 to 0.30) | |
Liquid container size | 0.38 (0.30 to 0.45) | N/Ad | N/A | 0.38 (0.30 to 0.45) | 0.35 (0.26 to 0.44) | 0.47 (0.33 to 0.59) |
aDD: disposable devices.
bRDD: reusable devices with disposable pods or cartridges.
cRDR: reusable devices with refillable pods, cartridges, or tanks.
dN/A: not applicable.
Although we found substantial to almost perfect agreement between photos and self-report for all variables measured in this study, agreement and ICC for nicotine concentration was substantially lower than those for other variables assessed. As previous research has identified [
In addition, we found the lowest agreement for device brand and model for participants who were using disposable devices and the highest agreement for those using reusable devices with disposable pods or cartridges. Despite finding the highest agreement for device brand and model among users of reusable devices with disposable pods or cartridges, agreement for liquid flavor was lowest among these users. These differences may be owing to low knowledge of these characteristics among users of certain device types or inaccurate or incomplete information about these characteristics on websites for certain device types. The wide variation in agreement for nicotine formulation across device types suggests a need for future research into how to evaluate the nicotine formulation of a liquid and how to ensure adequate labeling of nicotine formulation, so that participants’ self-reports can be more accurate. Given that the lowest agreement was observed in users of disposable devices and reusable devices with disposable pods or cartridges, it may be that users of nonrefillable devices tend to be less informed about the nicotine formulation of their liquid, possibly owing to poor labeling or lack of concern about nicotine formulation. In addition, these may be a result of challenges in finding accurate information about nicotine formulation for disposable devices and reusable devices with disposable pods or cartridges on websites. Customized liquids presented a unique challenge in assessing bottle size, with lower agreement than noncustomized liquids. This is likely because of challenges in coding the photos of customized liquids, as users sometimes refill bottles obtained from previous purchases of brand-name liquids but may be reporting the quantity of refill rather than the actual size of the bottle. It is also possible that users of customized liquids have submitted photos of a noncustomized liquid owing to concerns that we will be unable to use data from an unlabeled customized liquid bottle.
Krippendorff
Disagreement between self-reports and photos may also be caused by inaccuracies in website data used to code photos (information about device and liquid characteristics are sometimes inconsistent across websites), user modifications of devices in manners inconsistent with manufacturer intended use of the product (eg, some consumers refill JUUL pods, which are intended to be disposable [
It is also important to note that the number of excluded values differ between self-reports and photos. This is an important consideration in deciding on an approach as missing values can reduce the sample size or lead to a requirement for more resources for recruitment to obtain a sufficient sample size. When discounting self-reported values that were excluded owing to the skip logic of the survey, self-reports have more usable data for all variables except device model and nicotine formulation. Values were excluded based on the skip logic in our survey only to avoid asking unnecessary questions or questions to which users are unlikely to provide reliable responses. Users of disposable devices (119/1209, 9.84%) were not asked about their device refillability, adjustable power, presence of a device display, liquid brand, or liquid bottle size; users of reusable devices with disposable pods or cartridges (288/1209, 23.82%) were not asked about their liquid container size; and users of customized liquids (294/1209, 24.32%) were not asked the brand of their liquid. Although both self-reports and photos may produce fairly accurate results for device model and nicotine formulation, as indicated by the generally high agreement for these variables, photos may ultimately be superior to self-reports for device model and nicotine formulation because they capture more usable data.
The results outlined in this study provide valuable information about assessing several characteristics of e-cigarette devices and liquids. The agreement between photos and self-reports was substantial to very high for all variables included in this study except nicotine concentration. In addition, photos may provide more usable data for device model and nicotine formulation, whereas self-reports may provide more usable data for the remaining variables: device brand, device reusability, adjustable power, device display, device refillability, liquid brand, liquid flavor, nicotine concentration, and liquid bottle size (for users of refillable devices). However, using both of these approaches in tandem may allow for higher data quantity and quality, as values missing from one approach can be supplemented using the other (ie, values missing from photo data may be filled in using data from self-reports and vice versa) and data can be cross-checked between the 2 approaches.
The strengths of this study include the large sample size, which allows for analyses of subgroups of participants such as users of various device types. In addition, participants were not aware that they would be required to submit photos for the survey until after the self-report questions had been answered. Therefore, it is unlikely that the agreement data were affected as a result of participants being more accurate in their self-report owing to knowledge of the photo-uploading part of the study. However, the reverse may not be true; it is possible that some participants were more likely to upload photos of the device or liquid they self-reported because they had already reported information about the device or liquid.
Although the questions included in the survey were selected from or based on previously validated questions (eg, PhenX Toolkit) or established surveys (eg, Population Assessment of Tobacco and Health and International Tobacco Control survey) where possible, novel questions were created if a given characteristic had not been previously assessed or validated (eg, device reusability). In addition, the rigorous data review procedures in this study (eg, eliminating participants with very low knowledge of e-cigarette devices or liquids; photos from the internet; or photos with a brand that did not match the self-reported brand for JUUL, Voopoo, and Vaporesso and requesting a utility bill for participants who failed an initial identity authentication) may have resulted in a sample with a larger proportion of highly conscientious participants, and thus, higher agreement between self-reports and photos than we may otherwise have seen. JUUL, Voopoo, and Vaporesso were excluded from device brand analyses. As these were provided in the question prompt for brand as examples, bots and other invalid submissions frequently listed these as brand and submitted photos of other devices. These submissions were excluded from our analyses and, therefore, any valid self-reported JUUL, Voopoo, and Vaporesso submissions would be a match for photo brand. Therefore, the results for brand calculations cannot be extended to include these 3 brands. Results should be interpreted carefully with the understanding that “I don’t know” or missing responses were excluded from the percentage agreement, Krippendorff
e-Cigarette device and liquid data from both self-reports and user-submitted photos can present challenges. The high agreement between self-reports and photos suggests that these 2 methods yield data of similar accuracy for several variables: device brand, device model, device reusability, presence of adjustable wattage, presence of a display, device refillability, liquid brand, liquid flavor, and liquid bottle size. Self-reports provided a higher quantity of data than photos for all these variables except device model and nicotine formulation, for which photos provided a higher quantity of data. Although self-reports may be sufficient in certain studies and for specific variables, using these 2 approaches in tandem presents an opportunity to optimize the quality and quantity of data as it allows data to be cross-checked between 2 sources and provides an additional source when data are missing from one source. Further research is needed to understand how to assess nicotine concentration and other variables not included in these analyses (eg, wattage and resistance), how consistently and accurately the users of disposable devices and reusable devices with disposable pods or cartridges report on their liquid bottle size, the time and resource requirements for successful implementation of each of these 2 approaches (ie, photos and self-reports), and potential new innovative techniques for assessing e-cigarette device and liquid characteristics (eg, video recordings and daily e-cigarette use diaries).
intraclass correlation coefficient
Research Electronic Data Capture
The authors thank Katie Li and Vikas Vattipally for their contributions to participant recruitment, photo coding, and data cleaning for this study. They would also like to thank Virginia Commonwealth University’s Survey and Evaluation Research Laboratory for their work in data collection. Research reported in this publication was supported by National Institute on Drug Abuse and Food and Drug Administration Center for Tobacco Products under award number U54DA036105. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Food and Drug Administration.
EC conceived the research question. EC and JJH coded the data. EC, JJH, QN, JS, and KW cleaned the data. EC and QN conducted the data analysis. EC wrote the first draft of the manuscript. All authors contributed to the design of the study, critically reviewed the drafts of the manuscript, and read and approved the final manuscript.
None declared.