Our study is a cross-sectional TikTok post content analysis to identify important features associated with high user engagement in TikTok posts. All data collected and analyzed in our study are publicly available from the TikTok platform, and all results were reported in aggregate form to ensure user anonymity.

Our study has been reviewed by the University of Rochester Research Subjects Review Board (RSRB) office and was determined to meet federal and university criteria for exemption under Exempt Category 4 (Study ID: STUDY00009334). The RSRB determined that this study qualifies as secondary research involving existing data and, therefore, does not require informed consent. All publicly available TikTok videos included in the analysis were deidentified before use, and no compensation was provided to the TikTok users. Our study report followed the Strengthening and Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines [35].

TikTok short videos (less than one minute) in English related to e-cigarettes from the United States were searched and downloaded using the TikTok research application programming interface (API) using keywords related to e-cigarettes, including “e-cig,” “vaping,” “e-cigarettes,” “e-cigs,” “ecig,” “ecigs,” “electroniccigarette,” “ecigarette,” “ecigarettes,” “vape,” “vapers,” “vapes,” “e-liquid,” “ejuice,” “eliquid,” “e-juice,” “vapercon,” “vapeon,” “vapefam,” “vapenation,” “juul.” [36]. The TikTok API has a limitation of around 100 on the number of videos we can collect each month. In February 2024, through the TikTok API, a total of 1487 TikTok videos (from 1064 unique creators) related to e-cigarettes posted between January 2023 and January 2024 were collected and confirmed manually in this study.

The TikTok video feature list was derived primarily from our previous work—namely, our characterization of Instagram image features for e-cigarette content and our manual coding of e-cigarette videos on TikTok and YouTube [31-33,37,38]. We then reviewed sample TikTok videos to identify additional features that could affect engagement and, as a team, agreed on the final set of features for this study. The TikTok video features analyzed in this study included promotional content (including advertisements or promoting the e-cigarette use), celebrity endorsements, background setting, perceived sex (male or female, based on the observation), social events, young adult themes, lifestyle portrayals, e-cigarette devices (the presence of physical devices), smoking or vaping behaviors (the action of smoking or vaping regardless of the presence of physical e-cigarette devices), talking, singing, dancing, humorous or silly content, cartoons or animations, vaping tricks, and the use of emojis. We considered 2 cutting-edge large language models to extract video features from TikTok videos: GPT-4 [39] and Video Large Language Model Meta AI (Video-LLaMA-7B) [40]. Both models can understand the visual and auditory content in the video. Based on a reported AI labeling accuracy of 94% [41] and a prespecified 10% margin of error for the 95% CI, the single-proportion sample-size calculation indicated that 22 videos are required for our pilot evaluation of GPT-4 and Video-LLaMA-7B feature labeling. To examine which large language model performs better, we randomly selected 25 videos from the 1487 videos and manually labeled the proposed features by 2 human coders. Two human coders worked together side-by-side to watch these videos and label the defined video features. GPT-4 uses vision models to break down the visual component of a video frame by frame and detect objects, faces, text in the image, and other visual elements to help us identify features from the TikTok videos. In this study, to reduce data size and the demand for computational resources while still capturing meaningful information in the video, we adopted a frame sampling strategy, one image frame for every 15 frames, to represent the video [42]. In this study, the OpenAI API was used to interact with the GPT-4 model, and well-designed prompts were used to generate the desired and optimal output from the model. Table S1 in Multimedia Appendix 1 lists prompts used to automatically extract video features from the remaining 1462 TikTok videos using the GPT-4 API.

We used the GPT-4 API to categorize each video’s background into one of the following 6 groups based on the description of the background, including car, public space (club, studio, indoor and outdoor, medical building, restaurant, spacecraft, and other), private space (bathroom, bedroom, home, kitchen, room, and office), outside (beach, forest, garden, natural, space, campus, cave, urban, and natural), shop (shop and vape shop), and not detectable. Based on the description of the lifestyles from GPT-4, we classified the lifestyles into casual, leisure (fitness, gaming, and leisure), and no lifestyle (educational, health, industrial, vaping, working, or others) categories.

A linear mixed effects model with random intercept to account for multiple videos posted by the same user was used to model the association of TikTok user engagement with extracted features. We calculated the Spearman correlations among likes, shares, comments, and views of TikTok videos and found significant moderate-to-strong correlations across these metrics, as expected (Figure S1 in Multimedia Appendix 1). Given this interdependence, a more appropriate approach is to use overall engagement as the outcome variable, combining likes, shares, comments, and views using the formula applied by TikTok. The TikTok user engagement measure was calculated based on the formula [(number of likes+number of comments+number of shares)/(number of views+0.5)] X 100 [43]. For each video, add up all user actions (likes, comments, and shares) and then divide that total by the number of views. This gives interactions per view, so it reflects how actively viewers responded to the content, independent of how many people saw it. The value of 0.5 is added to the formula to minimize the generation of infinity values when the number of views is 0. Wilcoxon rank sum tests were used to compare the TikTok user engagement on a log scale between different feature categories. A linear mixed effects model was used to identify significant features associated with TikTok user engagement. The purposeful variable selection method was used to select important features for the final model, and Tukey’s method was used to control the familywise error rate for pairwise comparisons. Statistical analysis software R 4.3.1 (R Core Team, 2007) was used for the data analysis, with a significance level of 5% for all the analyses.

From the 1487 e-cigarette-related videos, we randomly selected 25 TikTok videos for hand-coding video features. We compared the accuracy in extracting features between 2 popular large language models, GPT-4 and Video-LLaMA-7B, using the hand-coding video feature as the golden standard. As shown in Table S2 in Multimedia Appendix 1, for the video features we selected, the large language model GPT-4 showed much higher accuracy (83% to 100%) than the model Video-LLaMA-7B (24% to 88%). Therefore, we used GPT-4 to extract features from the remaining 1462 videos.

Using the Wilcoxon sum rank test, we compared the distribution of TikTok user engagement measures on a log scale between different categories of those identified video features (Table 1).

As shown in Table 1, TikTok videos with a background in a car versus other background categories (5.48 for car vs 3.06‐4.65 for other background categories), featuring females versus males or females and males or no people (4.98 for females vs 3.63‐4.47 for males or females and males or no people), advertising social events (4.60 vs 4.13), including young adults (4.69 vs 3.92), showing casual lifestyles (4.70 vs 4.03-4.36), without the presence of physical e-cigarette devices (4.72 vs 3.93), having the action of people smoking or vaping (4.61 vs 4.35), including people talking (4.88 vs 3.80) or singing (5.31 vs 4.39), have significantly higher user engagement than TikTok videos without those features (all P<.05). TikTok videos without promotion content had substantially higher user engagement than TikTok videos with promotion content (4.66 vs 3.02). In addition, TikTok videos that are funny or silly (5.09 vs 4.16) or contain emojis (4.90 vs 4.26) had significantly higher user engagement than other TikTok videos (P<.001). TikTok videos with features like celebrity endorsement, people dancing, cartoons or animations, and vape tricks were not significantly different from TikTok videos without those features.

In the multivariate linear mixed effects model, several video features, such as lifestyle, dancing, and singing, were not significantly associated with TikTok user engagement. According to the purposeful model selection method, these were not included in the final linear mixed effects model. Results from the linear mixed effects showed significant features associated with the TikTok user engagement after adjusting for other variables in the model (Figure 1). As the exponential of the parameter estimates were the rate ratios (RR) from the linear mixed effects model, an estimated RR of 1 or higher was classified as high engagement, and an estimated RR smaller than 1 was classified as low engagement. TikTok videos featuring young people had significantly higher user engagement than those without (estimated RR=1.24, 95% CI 1.00‐1.53; P=.048), corresponding to 24% greater engagement. TikTok videos with people talking had significantly higher user engagement (RR=1.63, 95% CI 1.30‐2.05; P<.001), meaning user engagement is 1.63 times that of videos without people talking. TikTok videos that are funny or silly had significantly higher user engagement than TikTok videos that are not funny or silly (RR=1.61, 95% CI 1.29‐2.00; P<.001). TikTok videos that contain emojis had significantly higher user engagement than TikTok videos without emojis (RR=1.88, 95% CI: 1.48‐2.38; P<.001). There is a marginally significant association between TikTok videos with vaping tricks and TikTok user engagement (RR=1.62, 95% CI 0.97‐2.71; P=.07). On the contrary, TikTok videos containing promotional content had significantly lower user engagement than TikTok videos without promotional content (RR=0.60, 95% CI 0.45‐0.81; P=.001).

The background setting showed a significant association with engagement. Accordingly, we estimated all pairwise contrasts between video backgrounds using Tukey’s method. Pairwise comparisons showed significant differences in TikTok user engagement among different video backgrounds (Figure 2). TikTok videos with a background in the car had significantly higher user engagement than TikTok videos with a public space background (RR=3.91, 95% CI 1.25‐12.20; P=.009). Other video background comparisons showed no significant differences, as their 95% CIs included 1.

Our study is the first to investigate video features in e-cigarette-related TikTok videos associated with high TikTok user engagement. Using artificial intelligence (large language models such as GPT-4) and generalized linear models, we extracted and identified e-cigarette-related TikTok video features associated with high user engagement, such as background settings, young adult presence, funny or silly content, and emojis. Some of these identified features (such as background setting, humorous content, and proper emojis) could be used to design future e-cigarette prevention and cessation videos to increase user engagement.

In this study, as a proof of concept, we have tried to extract a set of features from e-cigarette-related TikTok videos. Some are general video features like the background, young people, and fun. In contrast, others are more specific to e-cigarette-related videos, such as smoking or vaping and vaping tricks. While Video-LLaMA is designed to analyze videos, our results showed that GPT-4 performed better than Video-LLaMA in identifying these video features. One potential reason for this is that GPT-4 is a multimodal model trained on a much more extensive training dataset. In addition, while Video-LLaMA might be more potent in understanding video motion and temporal dynamics, GPT-4 might perform better in semantic feature extraction at the frame level, especially from still frames in videos, by leveraging cross-modal information.

The pairwise comparison results within the linear mixed effects model framework showed the differences of various video background settings in their associations with TikTok user engagement. The reduced number of significant findings for the car background in the pairwise comparisons resulted from the stricter Type I error control applied via Tukey’s method. TikTok videos with backgrounds in a car had significantly higher user engagement than TikTok videos with backgrounds in public spaces. These results showed TikTok users prefer background settings in a more closed or private space than public spaces for e-cigarette-related TikTok videos. The reason for this might be related to the vaping ban in public places, including all bars and restaurants, which was recommended by the World Health Organization (WHO) in August 2016 and implemented in most United States states, including Alabama, California, and New York [44]. The widely adopted vaping ban in public places in the United States might lead people to view vaping in public spaces, such as bars, restaurants, or medical buildings, as inappropriate. Thus, e-cigarette-related TikTok videos in public places had less user engagement than TikTok videos with car backgrounds.

E-cigarette-related TikTok videos with young adults had higher user engagement. This might be due to the significant population of e-cigarette users being youth and young adults, and most of the TikTok users being young people. However, TikTok videos that showed only the physical e-cigarette device were less attractive to users—likely because they lacked human cues, offered little narrative value, and looked like advertisements. TikTok videos that are funny or silly attract more attention. Humor seems to be an effective tool to connect with audiences on TikTok by capturing users’ interest, encouraging sharing, and fostering a sense of community. Humor could also help increase the loyalty of followers [45]. E-cigarette-related TikTok videos featuring vaping tricks appear very impressive to viewers [37], sparking curiosity among many users about how those tricks are performed [31]. Our study found a positive association between vaping tricks and user engagement that was marginally significant. While vaping tricks are commonly used to promote vaping on social media, they should be appropriately regulated or controlled to reduce their influence on susceptible social media users. Emoji is another powerful tool that attracts TikTok users’ attention. In TikTok videos, emojis could serve as visual cues to express feelings, reactions, and attitudes to effectively communicate with viewers on an emotional level and attract their engagement [46,47].

Current policy is insufficient in preventing the spread of vaping promotion videos on TikTok [23]. Identifying video features linked to higher engagement in e-cigarette–related TikTok posts has clear implications for policy and practice. For public health communication, prevention content can ethically adopt engaging formats to improve reach—eg, featuring young adults, using light emoji overlays and humorous content, and filming in relatable settings such as cars—while avoiding elements that glamorize product use.

Our current study has several limitations. First, the e-cigarette-related TikTok video features included in this study are only a subset of video features. Other video features, such as the audio features, should be further investigated in future studies. Our study is the first one to investigate e-cigarette-related TikTok video features associated with social media user engagement. Here, we only tested some features that could be identified using current artificial intelligence techniques; for example, the current GPT model could not detect the background settings from some of the TikTok videos. With the rapid development of artificial intelligence techniques, we believe more features could be accurately labeled by the GPT models or other more advanced models, which can be included in our future studies. Second, our list of TikTok video features may not be applicable to other social media platforms. Therefore, it is crucial for future studies to explore important features on different social media platforms, such as Instagram and YouTube. Third, the accuracy of some feature labeling by the current GPT models has not reached 100% yet. There might be mislabeling of some features in our data, which might lead to some bias in our results. The accuracy of video features could be further improved with the availability of more advanced large language or video models. In addition, because sex was inferred from perceived physical characteristics and presentation in the videos rather than self-reported identity, the measure reflects perceived sex and is subject to misclassification bias, with potential exclusion or misrepresentation of transgender, non-binary, and gender-diverse individuals. Fourth, due to the unavailability of data, we cannot determine the demographics of TikTok users who were engaged with these e-cigarette-related videos, which should be investigated using other approaches. Moreover, we do not have the information about account-level factors—such as the number of unique accounts reacting, whether an account is personal versus sponsored or brand, and whether a post was boosted or advertised—that may also influence engagement. Fifth, not all features identified in this study might be appropriate for vaping prevention videos, which should be further tested and validated in future studies. Sixth, the data collection was conducted in February 2024 and included TikTok videos posted from January 2023 to January 2024. The numbers of views, likes, shares, and comments may change over time. Although the moderate-to-strong correlations across views, likes, shares, and comments imply that the engagement ratio (likes+shares+comments)/views should remain broadly consistent, potential bias could be introduced with the changes in the engagement metrics over time. Seventh, because the TikTok API limits monthly retrievals, our 1-year sample included 1487 videos. Larger samples in future work would increase power to detect additional features associated with user engagement. Eighth, 2 coders annotated video features side-by-side rather than independently; consequently, we could not quantify inter-rater reliability (eg, Cohen Kappa). This consensus approach may inflate agreement and introduce shared-observer bias. Finally, with the rapid changes in e-cigarette marketing and social media, it is necessary to closely monitor and update new developments of social media posts with high user engagement.

In this study, we showed that specific TikTok video features, including background settings (eg, car), talking, funny or silly, and use of emojis, were significantly associated with higher TikTok user engagement. These findings offer practical guidance for designing more engaging vaping-prevention videos for a broader reach of social media users.