This methodological, multiphase study comprises 5 main stages to develop the LassVQ scale for assessing the HESVs on social media. This study was reported in accordance with COSMIN (Consensus-Based Standards for the Selection of Health Status Measurement Instruments) reporting guideline [27]. A completed COSMIN checklist is provided in Checklist 1.

This study was approved by the Biomedical Ethics Review Board of Peking University (IRB00001052-24115) and complied with the Declaration of Helsinki principles. All participants provided informed consent. Participants were assured that their privacy and confidentiality would be strictly protected, and all data were anonymized before analysis. No personal identifying information was collected or stored. No compensation was provided to participants.

To develop an initial item pool of evaluation indicators, a literature review was conducted by predefined search terms, including “video,” “social media,” “social networking,” YouTube, TikTok, Douyin, quality, reliab*, understandability, actionab*, credib*, interactiv*, comprehensive*, accuracy, completeness, and clarity. Systematic searches were carried out across 5 major databases—PubMed (n=1864), Web of Science (n=6191), Embase (n=2817), CNKI (n=3543), and WANFANG (n=411) databases, spanning from the inception of each database to May 2024. Eligible records were limited to those published in English or Chinese and included peer-reviewed journal articles, academic reports, and conference proceedings relevant to the development or application of evaluation indicators. After removing duplicates (n=5401), 9425 records were screened based on titles and abstracts. Records deemed irrelevant or inaccessible, or that did not meet the inclusion criteria (n=9119), were excluded. The full texts of the remaining 306 articles were reviewed in detail, resulting in 72 studies meeting the final inclusion criteria. The research was conducted by 2 researchers (YH, YY) who participated in the literature search and screening. The entire research process was supervised and guided by one professor (JS). The professor was engaged in health education research for many years. The complete list of included studies is provided in Appendix S1 in Multimedia Appendix 1.

The study group summarized relevant research evidence, including preliminary research of literature review, brainstorming undertaken in the study group, literature reviews of the tools to assess printed health education materials. Then, a structured brainstorming session was subsequently organized with the study group, which included both senior experts and younger technical professionals. During the meeting, participants reviewed the draft framework and items, suggested additional items, identified potential overlaps, and proposed modifications. All inputs were recorded in real time by a designated notetaker (YY) to ensure completeness and accuracy. The resulting item pool integrated evidence from the literature (as the foundation) and expert opinions (to contextualize and refine items). Based on the 5W model, the initial version of the evaluation system comprised 5 primary indicators and a total of 119 indicators.

A pilot study with 15 participants (9 viewers and 6 producers of HESVs) was conducted to evaluate readability and comprehensibility of the draft scale, following the NGT. The main study did not include these participants. Data were collected through face-to-face interactions to ensure detailed feedback on the scale items during the pilot phase. Participants rated each item using a 5-point Likert scale (1=very poor, 5=excellent) for both readability and comprehensibility. Open-ended feedback was also solicited to identify potential ambiguities. The subsequent survey was conducted with the main participants after adjustments were made based on feedback received during the pilot test.

The overall process of LassVQ development and validation is summarized in Figure 1. This flowchart provides an overview of each step, which is described in detail in the following sections.

Based on the preliminary literature review, a framework was established, including defined dimensions, operational definitions, and an initial item pool. The brainstorming session resulted in the generation of 2 new items and the modification of 3 overlapping items. The final initial version of LassVQ consisted of 5 primary indicators, 20 secondary indicators, and 94 tertiary indicators as illustrated in Appendix S5 in Multimedia Appendix 1.

This study adopted 2 rounds of NGT meetings, enrolling a total of 11 experts. Demographic characteristics of the expert panel are outlined in Table 1. Among the selected experts, 2 were male and 9 were female, 45% (5/11) of experts had a doctoral degree. The panel represented a multidisciplinary background, comprising 2 health educational video practitioners, 5 health education researchers, 3 health education practitioners, and 1 library informatics researcher. Furthermore, 80% (9/11) of them ranked associate professor or professor, with 55% (6/11) having more than 20 years of research and work experience. All participating experts were from institutions located in Beijing. The Cs of each key indicator of the NGT was 0.77. The average value of the Ca and Cr was 0.93 and 0.85, respectively. The Kendall W of the revised indicator system is 0.216 (χ²₄₉=105.8, P<.001), as presented in Table 2, while our predefined consensus threshold (≥80% experts agreement) was achieved, indicating that consensus could be reached among the experts.

In round 1, 11 experts suggested amendments to some indicators. During the discussion, there were 5 primary indicators and 20 secondary indicators, with the number unchanged. The primary indicator, “communication target,” did not reach a consensus. For secondary indicators, 7 of 20 did not reach consensus, which will be addressed in round 2 of NGT. The “reliability” of the primary indicator “communicator” was changed to “credibility.” Tertiary indicators were screened as follows: (1) revised “reasonable and clear navigation structure” to “reasonable navigation structure,” and revised “the comments are positive” to “the positive ratio of comments;” (2) added “views volume;” (3) two indicators “available and easy to use forwarding function” and “available and easy to use like function” were merged into 1 indicator “available and easy to use like and share functions,” and “available and easy to use comments and messages” and “available and easy to use messaging function” were merged into 1 indicator “available and easy to use comments and messages functions.”

In round 2, the expert panel achieved consensus about the final indicator system, which then deleted 1 primary indicator, 7 secondary indicators, and 59 tertiary indicators based on the consensus criteria. The medication and consensus rate of indicators of the 2 rounds of NGT were presented in Appendices S6 and S7 in Multimedia Appendix 1. After the 2 rounds of NGT, 4 primary indicators, 13 secondary indicators, and 34 tertiary indicators finally reached consensus, as presented in Figure 1.

According to the AHP, the quality evaluation system of HESVs on social media was divided into 4 layers. The 9-point intensity of relative weight of importance scale and random index is illustrated in Appendices S2 and S3 in Multimedia Appendix 1. Matrix and weight analysis of the primary indicators (communicator, communication content, communication channel, and communication effect) are presented in Table 3. Among the 4 primary indicators of the evaluation system, “communication content” was perceived as the most influential variable and had a weight of 45.73%, more important than “communicator,” “communication channel,” and “communication effect,” with a relative contribution of 34.58%, 9.93%, and 9.81%, respectively.

The relative weights of secondary indicators are presented in Appendix S8 in Multimedia Appendix 1. The secondary indicators analysis showed that in the “communicator,” “credibility” was perceived as more important than “authoritative” (within-dimensional weight=56.67% and 43.33%, respectively). Among the “communication content,” the relative importance was “scientific” (24.26%), followed by “understandability” (18.59%), “reliability” (18%), “actionability” (14.64%), “watchability” (12.99%), and “attractiveness” (11.53%). In the “communication channel”, “availability” (74.62%) was viewed as more important than “interactivity” (25.38%). Among “communication effect,” the relative importance was “social attention” (39.89%), followed by “perceived usefulness” (36.15%) and “communication engagement” (23.96%).

The relative weights of tertiary indicators are presented in Appendix S8 in Multimedia Appendix 1. The ten tertiary indicators with highest ranks were: (1) recommended as a trending video; (2) positive comments ratio, tied for (3) content’s consistency with scientific common sense, and (4) content complies with ethical standards; (5) content contains necessary relevant information; (6) stable and effective video information source; (7) motivation to share; (8) behavioral guidance or operation techniques are presented with clear steps in logical order; (9) expression is understandable, avoiding medical terms; and (10) theme is clear. The top 1, 2, and 7 tertiary indicators fell into the “communication effect” dimension; numbers 3, 4, 5, 8, 9, and 10 into the “communication content” dimension.

The content validity index for the relevancy of the LassVQ evaluation system was calculated to be 0.87 (Table 4). The kappa designating agreement of relevance of the LassVQ evaluation system was all above 0.74 (κ=0.80‐1.14).

A total of 15 participants completed the pilot questionnaire, including 9 viewers (aged 19‐46 years) and 6 producers (aged 33‐54 years). Participant characteristics are detailed in Appendix S9 in Multimedia Appendix 1.

The mean readability score across all items was 4.87 (SD 0.26), and the mean comprehensibility score was 4.92 (SD 0.18), indicating that the items were generally clear and understandable. Qualitative feedback highlighted minor issues in wording and phrasing, which were addressed through targeted revisions. For example, vague or informal phrases such as “content has info that’s kind of needed and related stuff” were replaced with clearer wording like “content contains necessary relevant information.” Similarly, ambiguous expressions like “expression uses unclear or confusing language” were revised to “expression is understandable, avoiding medical terms.” These refinements improved the clarity, professionalism, and user-centeredness of the final scale.

This study developed LassVQ, an HESVs evaluation system, focusing on 4 key dimensions: communicator, content, channel, and effect, with credibility, scientific rigor, accessibility, and social engagement as core quality indicators. The indicators showed preliminary evidence of acceptable content validity. LassVQ is a preliminary tool that requires further empirical validation. It may, in the future, contribute to supporting researchers in assessing HESVs on social media, aid platforms in video screening, and assist policymakers in developing video production standards.

Valid and reliable measurement tools play an important role in supporting quantitative research and are also frequently used in communication studies [33]. With the emergence and popularity of HESVs on social media and concerns about their quality, it is of both theoretical and practical meaning to explore a quality evaluation system [34]. Based on the 5W model, this research developed the LassVQ evaluation system using the nominal group technique and the analytic hierarchy process. Overall, results of Cs, Ca, and Cr suggested that experts are familiar with the topic, with judgments influenced by practical experience and theoretical analysis and demonstrated a relatively high authority. The result of the modified kappa suggested that the LassVQ evaluation system possesses acceptable content validity [32].

Shawahna [35] and Hong et al [36] used to combine NGT and AHP and facilitated decision-making and quantitatively observe and reviewed in health care dilemmas and streams’ quality, respectively. The NGT calls for methods to include every expert’s opinion. As would be expected, some experts were more vocal than others, and so sessions would include a “round-robin,” which means asking each participant to speak in turn in the order in which they are sitting to ensure each had suggested discussion. The anonymous voting process was intended to encourage participant engagement. The participating experts included both health education researchers and practitioners, along with HESV practitioners and a library informatics researcher. All experts were involved in aspects of HESV production and communication. This combination with AHP allows for analyzing an expert’s subjective judgment with mathematical form and conducting multiobjective decision-making analysis of the scientific treatment to ensure the scientific result [37]. Experts adopt judgment matrices to determine the importance value of each indicator, and the result shows that the consistency of the pairwise comparison matrices at all levels met the consistency test criterion, with CR<0.10 [37], suggesting acceptable logical consistency and supporting scientific rigor and reliability. The weighted value of the primary indicator was in the following order: communication content (0.46), communication effect (0.34), communication channel (0.10), and communicator (0.10). Existing tools were designed for printed materials, developed many years ago, and may not be fully applicable to the social media context. Furthermore, DISCERN consists of 6 questions focusing on diseases and treatment, while the JAMA benchmarks take advertisement into consideration.

The Lasswell’s 5W communication model, a classic theory in information communication, offers a theoretical framework for understanding the dissemination process of HESVs on social media [25]. It was put forward in 1948 and still has its applicability to this day, which divides the communication process into 5 parts, including the communicator, information, channel, audience, and effect. Critics argue that the 5W model’s linear approach is outdated and insufficient for adapting to the dynamic environment of the internet [38]. However, consistent with the point by Sapienza et al [39], this study holds that the 5W model is flexible and we should dialectically explore new applications of it. For instance, Wei et al [24] expanded the 5W model into an 8-element communication framework for internet health rumors, while Xu et al [40] designed a questionnaire based on the 5W model to investigate caregivers’ needs regarding medication content and dissemination channels. In the 2 rounds of NGT discussions and indicator evaluations, this study removed the audience dimension from the original 5W model and restructured the dimensions into 4 categories: communicator, communication content, communication channel, and communication effect. Experts concurred that within the social media context, the roles of communicator and audience often overlap, with individuals potentially fulfilling both roles simultaneously [25]. Since audience research typically examines viewer characteristics and responses, while our study focuses on the quality of HESVs, including this dimension was considered outside our methodological scope.

Following the NGT discussions, the study excluded the communicator’s influence, originally measured by tertiary indicators such as the communicator’s total number of followers, total number of received likes, and total number of posted videos. Bengi et al [41] advised focusing on the communicator’s credibility and professional background rather than metrics like follower count or video quantity. The elaboration likelihood model indicated that professional titles may serve as signals of expertise, potentially enhancing viewer engagement [42,43]. The influence of the communication channel was also removed, including metrics such as “number of platform users,” “frequency of platform being forwarded by mainstream media,” and “the proportion of health popularization videos on the platform.” While platforms like TikTok (ByteDance) garner significantly more views and likes than YouTube, they have comparatively fewer high-quality videos [41]. The model also eliminated breadth-of-spread metrics, such as “balanced gender of users,” “balanced age distribution of users,” “wide distribution of users,” and “balanced classification of the user’s city.” The NGT experts argued that these indicators were considered of limited significance and that data availability was relatively low. When producing or evaluating HESVs, priority should be given to the content’s theme and target audience [15,44]. For example, videos about menstruation inherently tend to attract an audience with an imbalanced gender and age distribution. For “communication recognition” and “communication participation,” NGT experts emphasized the significance of views as a direct communication metric. They also suggested combining views, likes, collections, and reposts as a single metric, which aligns with the research by Erdem and Karaca [45] on the video power index used for evaluating YouTube content.

For the tertiary indicators of the understandability of communication content, the NGT removed some indicators deemed less relevant. Among those addressing the consistency between theme and title, only “theme is clear” was retained as the most representative indicator [46]. Indicators like informative subtitles, clear summaries, 3 to 5 core points, and highlighting of core information were also removed, as they were deemed unsuitable in the context of short social media videos [18]. This study refined the tertiary indicators for content attractiveness by focusing on the “three Vs” of communication—verbal, vocal, and visual [16], a model recommended for enhancing educational outcomes in HESVs [18]. Experts noted that the indicator “subtitles are consistent with dubs” was inappropriate since subtitles might be autogenerated text. Indicators of actionability were narrowed down to emphasize logical presentation and clear guidance on how to take action, consistent with the study by Sui and Zhang [47] on the role of logical coherence in changing attitudes. All indicators related to perceived cultural appropriateness were excluded, as experts argued that evaluating HESVs based on target audience and linguistic preferences might lead to conflicting conclusions.

This study is the first to develop a comprehensive evaluation system for HESVs on social media (LassVQ). Some limitations of this study should be noted. The NGT itself is less commonly used than the Delphi process in health communication, with disadvantages including limited rounds of voting [48]. However, this is outweighed by allowing face-to-face discussions and anonymous voting, which yield higher response rates than would have been achieved with mailed questionnaires [49]. Further research is required to confirm the validity of the findings obtained here on this evaluation system by applying it to HESVs on social media platforms. For the pilot study, the small sample size limits the generalizability of the findings related to readability and comprehensibility. As an exploratory step, the pilot was primarily intended to guide item refinement, and its results should be interpreted with caution. The expert selection process, constrained by the authors’ available resources, may introduce certain limitations and potential biases. The other limitation of the study lies in its focus on short-form video content, which may limit the applicability of the findings to other types of media, such as long-form videos or text-based health information. Although short videos are highly popular on social media, the evaluation criteria developed may not fully capture the nuances of different content formats. Future research could explore how the evaluation system might need to be adapted or expanded for other forms of health communication content across various platforms. The LassVQ evaluation system may offer a useful framework for analyzing the current circumstance of HESVs on social media, although further application and validation are needed to confirm its broader utility.

A more detailed investigation into the key factors for the quality improvement of HESVs on social media is needed to clarify. To support external validation, several health video producers and viewers will apply the LassVQ to a representative sample of high-view HESVs from major social media platforms. Intraclass correlation coefficients will be used to evaluate interrater reliability. Videos will also be rated using DISCERN and Global Quality Score, enabling correlation analyses to examine LassVQ’s predictive accuracy.

In summary, the quality evaluation system for HESVs on social media (LassVQ) developed in this study offers a structured approach to enhancing the quality of health communication throughout the entire video production and dissemination process. The system equips health care professionals with a research-based tool for the design, production, and evaluation of HESVs, with an emphasis on critical quality indicators. Future research could expand on these findings by refining the evaluation criteria, integrating additional perspectives from health care professionals, or tailoring the system to address specific health topics. The application of the LassVQ enables researchers and practitioners to continually enhance the effectiveness and impact of health education content disseminated on social media platforms.