This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
TikTok was an important channel for consumers to access and adopt health information. But the quality of health content in TikTok remains underinvestigated.
Our study aimed to identify upload sources, contents, and feature information of gallstone disease videos on TikTok and further evaluated the factors related to video quality.
We investigated the first 100 gallstone-related videos on TikTok and analyzed these videos’ upload sources, content, and characteristics. The quality of videos was evaluated using quantitative scoring tools such as DISCERN instrument, the Journal of American Medical Association (JAMA) benchmark criteria, and Global Quality Scores (GQS). Moreover, the correlation between video quality and video characteristics, including duration, likes, comments, and shares, was further investigated.
According to video sources, 81% of the videos were posted by doctors. Furthermore, disease knowledge was the most dominant video content, accounting for 56% of all the videos. The mean DISCERN, JAMA, and GQS scores of all 100 videos are 39.61 (SD 11.36), 2.00 (SD 0.40), and 2.76 (SD 0.95), respectively. According to DISCERN and GQS, gallstone-related videos’ quality score on TikTok is not high, mainly at fair (43/100, 43%,) and moderate (46/100, 46%). The total DISCERN scores of doctors were significantly higher than that of individuals and news agencies, surgery techniques were significantly higher than lifestyle and news, and disease knowledge was significantly higher than news, respectively. DISCERN scores and video duration were positively correlated. Negative correlations were found between DISCERN scores and likes and shares of videos. In GQS analysis, no significant differences were found between groups based on different sources or different contents. JAMA was excluded in the video quality and correlation analysis due to a lack of discrimination and inability to evaluate the video quality accurately.
Although the videos of gallstones on TikTok are mainly provided by doctors and contain disease knowledge, they are of low quality. We found a positive correlation between video duration and video quality. High-quality videos received low attention, and popular videos were of low quality. Medical information on TikTok is currently not rigorous enough to guide patients to make accurate judgments. TikTok was not an appropriate source of knowledge to educate patients due to the low quality and reliability of the information.
Gallstone disease is a common disease worldwide. It occurs in 10%-15% of adults in the United States and Europe [
The unprecedented growth of web-based medical information has significantly changed the way people obtain health information. Nowadays, more and more patients seek information online before seeing a doctor [
Although health content has been widely studied on video sites such as YouTube, it remains underinvestigated in emerging short-video apps such as TikTok [
No clinical data, human specimens, or laboratory animals were involved in this study. All information used in this study was obtained from publicly released TikTok videos, and none of the data involved personal privacy. In addition, the study did not involve any interaction with users; therefore, no ethics review was required.
In this cross-sectional study, the Chinese version of TikTok was searched on July 18, 2020, using the term gallstone in Chinese. Videos were sorted by comprehensive ranking with release time set to unlimited, which is the TikTok default. Non-Chinese videos were excluded until the top 100 videos in Chinese were displayed. We limited our analysis to the top 100 videos because multiple studies have confirmed that videos beyond the top 100 had no significant impact on the analysis [
Video sources were categorized as follows: (1) doctors, (2) individuals (ie, nonmedical professionals), (3) news agencies (ie, network media, newspaper, TV station, and radio station), and (4) organizations (ie, hospitals, health authorities, research groups, universities, or colleges). The content was categorized as follows: (1) surgery technique, (2) disease knowledge, (3) lifestyle, and (4) news. This classification allows us to group videos with the same content as much as possible, while distinguishing videos with different content.
The quality of the information in videos was assessed using the DISCERN instrument, the Journal of American Medical Association (JAMA) benchmark criteria, and Global Quality Scores (GQS). The DISCERN instrument was developed for judging the quality of health information on treatment choices [
All authors were senior surgeons engaged in hepatobiliary and pancreatic surgery for years and were knowledgeable in diagnosing and treating gallstone diseases. On the process for screening and rating, authors FS and JW used the DISCERN instrument, JAMA benchmark criteria, and GQS to evaluate the videos contemporaneously. The scores were determined through discussions. An arbitrator (SZ) resolved inconsistent scores between viewers and gave the final scores. Subsequently, all authors agreed on all the ratings.
The Journal of American Medical Association (JAMA) benchmark criteria.
Criteria | Description |
Authorship | Authors and contributors, their affiliations, and relevant credentials should be provided. |
Attribution | References and sources for all content should be listed clearly, and all relevant copyright information noted. |
Currency | Website ownership should be prominently and fully disclosed, as should any sponsorship, advertising, underwriting, commercial funding arrangements or support, or potential conflicts of interest. |
Disclosure | Dates that content was posted and updated should be indicated. |
Description of the Global Quality Score (GQS) 5-point scale used to evaluate videos with gallstone information on TikTok.
GQS | Description |
1 | Poor quality; poor flow of the site; most information missing; not at all useful for patients |
2 | Generally poor quality and poor flow; some information listed but many important topics missing; of very limited use to patients |
3 | Moderate quality; suboptimal flow; some important information is adequately discussed but others poorly discussed; somewhat useful for patients |
4 | Good quality and generally good flow; most of the relevant information is listed, but some topics not covered; useful for patients |
5 | Excellent quality and excellent flow; very useful for patients |
Means and SDs were used for descriptive statistics. The Kruskal-Wallis test assessed differences between groups, and the Dunn multiple comparisons test for two-way intergroup comparisons of quantitative variables with no normal distribution. Spearman correlation analysis was used to evaluate the relationships between quantitative variables.
The initial 100 videos collected by the TikTok search had 520,805 likes, 17,947 comments, and 181,355 shares. The mean video duration was 49.43 (SD 24.63) seconds. Moreover, the average number of days after uploading (days since upload) was 152.0 (SD 146.4) days by the data collection date.
Descriptive statistics for TikTok videos of different sources and content.
Variables | Likes, mean (SD) | Comments, mean (SD) | Shares, mean (SD) | Days since upload (days), mean (SD) | Duration (seconds), mean (SD) | |
|
||||||
|
Doctor (n=81) | 2247 (6006) | 174.9 (358.8) | 785 (2668) | 133 (122.5) | 52.73 (21.53) |
|
Individuals (n=10) | 2520 (3477) | 258.4 (341.7) | 311.3 (453.7) | 208.9 (216.9) | 37 (35.16) |
|
News agencies (n=6) | 52,232 (73,964) | 182.5 (106.9) | 19,070 (38,295) | 349.3 (189.8) | 28.17 (14.82) |
|
Organizations (n=3) | 73.67 (79.32) | 34.33 (40.46) | 79 (47.51) | 79.33 (66.11) | 44.33 (52.54) |
|
||||||
|
Surgery technique (n=21) | 657.7 (692) | 79.71 (82) | 68.95 (103.8) | 131.8 (115.6) | 58.24 (28.96) |
|
Disease knowledge (n=56) | 1655 (3394) | 177.3 (328.1) | 543.8 (2203) | 141.2 (134.6) | 46.18 (19.6) |
|
Lifestyle (n=16) | 5827 (11,657) | 292.5 (565.4) | 2204 (4168) | 149.8 (166.5) | 56.31 (32.09) |
|
News (n=7) | 45,871 (69,610) | 238 (204) | 16,313 (35,711) | 303.9 (211.1) | 33.29 (17.76) |
The mean DISCERN, JAMA, and GQS scores of all 100 videos are 39.61 (SD 11.36), 2.00 (SD 0.40), and 2.76 (SD 0.95), respectively. We found that JAMA was unable to make a discriminating and accurate assessment of video quality; the majority of the videos (93/100, 93%) were rated as 2, and others were rated as 1 (3/100, 3%), 3 (1/100, 1%), and 4 (2/100, 2%), respectively. Therefore, JAMA was excluded in the subsequent video quality and correlation analysis.
We compared the video quality based on different sources and contents with DISCERN and GQS scores. Three sections and the total score of DISCERN were analyzed. In section 1, DISCERN score of doctors was significantly higher than those of individuals and news agencies (
In GQS analysis, no significant differences were found between groups based on different sources or different contents (
According to DISCERN and GQS, gallstone-related videos’ quality score on TikTok is not high, mainly at fair (43/100, 43%) and moderate (46/100, 46%). We put these 5 levels in one-to-one correspondence, and the 5-level scores are reasonably consistent in DISCERN and GQS (
DISCERN scores for TikTok videos of different sources (a, b, c, d) and contents (e, f, g, h). *
Global Quality Scores analysis for TikTok videos of different sources and content.
The 5-level scores of DISCERN and Global Quality Scores (GQS; n=100).
Scores | Value, n (%) | |
|
||
|
16-26 (very poor) | 16 (16) |
|
27-38 (poor) | 24 (24) |
|
39-50 (fair) | 43 (43) |
|
51-62 (good) | 17 (17) |
|
63-80 (excellent) | 16 (16) |
|
||
|
1 (poor) | 14 (14) |
|
2 (generally poor) | 18 (18) |
|
3 (moderate) | 46 (46) |
|
4 (good) | 22 (22) |
|
5 (excellent) | 0 (0) |
The 5 levels of DISCERN and Global Quality Scores (GQS).
Detailed ratings of 100 gallstone videos by the DISCERN questionnaire with 16 questions.
Spearman correlation analysis revealed that the following variables were correlated positively: likes and comments (
DISCERN scores and video duration were positively correlated (
The relationship level between video variables.
Variable and analysisa | Likes | Comments | Shares | Days since upload | Video duration | ||||||
|
|||||||||||
|
1 | —b | — | — | — | ||||||
|
— | — | — | — | — | ||||||
|
|||||||||||
|
0.26 | 1 | — | — | — | ||||||
|
.01c | — | — | — | — | ||||||
|
|||||||||||
|
0.74 | 0.24 | 1 | — | — | ||||||
|
<.001d | .01c | — | — | — | ||||||
|
|||||||||||
|
0.26 | 0.08 | 0.21 | 1 | — | ||||||
|
.009e | .45 | .04c | — | — | ||||||
|
|||||||||||
|
–0.18 | –0.06 | –0.15 | –0.02 | 1 | ||||||
|
.07 | .54 | .13 | .87 | — |
aPearson correlation analysis and
bNot applicable.
c
d
e
Pearson correlation analysis between video quality scores and video variables.
Variable and analysis | DISCERN | GRSa | ||
|
||||
|
–0.30 | 0.06 | ||
|
.003b | .55 | ||
|
||||
|
–0.15 | 0.12 | ||
|
.14 | .24 | ||
|
||||
|
–0.23 | –0.04 | ||
|
.02c | .66 | ||
|
||||
|
–0.06 | 0.06 | ||
|
.55 | .54 | ||
|
||||
|
0.59 | –0.01 | ||
|
<.001d | .91 |
aGRS: Global Quality Scores.
b
c
d
In this cross-sectional study, feature information about gallstone videos on TikTok was analyzed at one time point, and the quality of videos was evaluated using DISCERN and GQS instruments. The majority (81/100, 81%) of gallstone videos on TikTok were posted by doctors, ensuring the professionalism of the content. However, according to DISCERN and GQS video quality ratings, the doctors’ videos were not as good as expected. It is worth noting that TikTok does not authenticate the bloggers’ identities who claim to be doctors, so there may be some possibility that the identities are fraudulent, such as impersonating doctors to gain extra video views.
The number of likes, comments, and shares reflect, to some extent, the popularity of a video or post [
The mean DISCERN, JAMA, and GQS scores of all 100 videos are 39.61 (SD 11.36), 2.00 (SD 0.40), and 2.76 (SD 0.95), respectively. Gallstone videos on TikTok were mainly evaluated at fair and poor grading ranges, consistent with previous studies about video quality on YouTube [
A positive correlation between video duration and video quality was found. The longer the video duration, the higher the video quality, which is the same conclusion as some previous studies [
We found a negative correlation between likes and DISCERN scores, as shown in
In addition, the results may be related to the user characteristics of TikTok. As TikTok is a lifestyle entertainment app, its users prefer entertaining videos. Videos with beautiful graphics are more attractive. However, videos with high credibility are not popular. Because professional content tends to be unentertaining or even dull and therefore challenging to attract an audience, it is harder for them to gain popularity.
Other studies have found that viewers may be more likely to seek video content that deviates from conventional treatment regimens [
We found that the JAMA score could not accurately assess the video information because there were only 4 indicators, and they were not precise enough. This result was consistent with previous reports [
The likes and comments of individuals and news agencies are significantly higher than those of doctors and organizations, indicating that individuals and news agencies have more significant influence than doctors and organizations on TikTok, but the quality of videos is lower than that of professionals. On the one hand, news and individuals need to improve the quality of their information. On the other hand, doctors need to increase their influence. Medical video creators need to avoid boring content on the premise of professional content and better attract audiences to further spread the correct medical information.
Due to the complexity and professionalism of medical content, the length of medical-related videos should be extended accordingly to ensure sufficient information and content quality. TikTok should give professional certification to some video creators who publish medical professional content. A certification mark can be given, increasing the audience’s recognition of the information and helping the dissemination of professional information. At the same time, a professional review can improve the quality and reliability of medical professional videos. When searching and evaluating medical content on the internet, health searchers mostly choose the results that appear on the first page of the search engine and rarely view the results that appear outside the second search results page [
In this study, no advertising content was found in the search results for gallstones. Since TikTok mainly features entertainment videos, medical videos are still in the process of increasing. The same goes for advertising videos.
This study is the first to use multiple tools (DISCERN, GQS, and JAMA) to evaluate the quality and reliability of gallstone-related videos on TikTok. We further analyzed the correlation between video duration, likes, comments, and shares and video quality, and found that likes and shares were negatively correlated with video quality, and video duration was positively correlated with video quality. This study is also the first to analyze the quality of surgical disease-related videos on TikTok.
However, our study still has some limitations. Because these 3 evaluation tools were initially designed to evaluate textual content, there remain some limitations to the content of the video, such as not being able to evaluate the sophisticated production level and audio effect of the video. Furthermore, although we conducted a thorough analysis of the number of comments, we did not conduct a deeper analysis of the content of the comments. Sometimes, viewers leave comments not as a praise, but due to dissatisfaction. These negative reviews also increase the number of comments. Of course, we found in our research that the number of comments and likes, as well as the number of comments and shares, are positively correlated, so the number of comments can to some extent reflect the popularity of the video. Therefore, conducting a more in-depth study of positive and negative reviews in comments to explore their relationship with video quality is something we need to further analyze in future research. The current videos on TikTok are mainly for entertainment, and medical videos account for only a tiny part; therefore, further observation is needed to study the evolution process of medical videos on TikTok.
Although videos on gallstones on TikTok are mainly provided by doctors and contain disease knowledge, they are of low quality. We found a positive correlation between video duration and video quality. High-quality videos received low attention, and popular videos were of low quality. Medical information on TikTok is currently not rigorous enough to guide patients to make accurate judgments. TikTok was not an appropriate source of knowledge to educate patients due to the low quality and reliability of the information.
The DISCERN instrument questionnaire.
chronic obstructive pulmonary disease
DISCERN instrument
Global Quality Scores
Journal of American Medical Association
The study is supported by the grant from Health Commission of Zhejiang Province (grant number JBZX-202004).
None declared.