A comprehensive literature search strategy was first developed to evaluate the cost-effectiveness analyses of DHIs for improving the uptake of vaccination programs. A preliminary search was conducted to identify potentially relevant digital technology search terms, such as “social media,” “social network,” “digital intervention,” “email,” and “mobile phone.” To construct the search strategy, we combined the search terms for “vaccination, vaccine, immunization,” “digital technologies,” and “cost-effectiveness analysis.” The following databases were searched (up to October 2022): MEDLINE (Ovid), Embase (Ovid), APA PsycINFO (Ovid), Web of Science, Scopus, CINAHL Ultimate (EBSCOhost), Center for Review and Dissemination, and Institute for IEEE Xplore. In addition, to verify retrieval completeness, a manual search on the reference lists of both included studies and associated systematic reviews was performed. This systematic review has been registered on PROSPERO (registration ID: CRD42023400329). The details of search strategy performed for each database and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 checklist are provided in Multimedia Appendix 1.

Full-text English-language health economic studies that met the following inclusion criteria were included: (1) adult or pediatric vaccination programs; (2) interventions delivered through digital technology; (3) full-scale health economic analyses including cost-effectiveness, cost-utility, cost-benefit, or cost-consequence analyses; and (4) evaluations conducted by model-based (including Markov model, decision tree model, discrete event simulation, and dynamic transmission model) or trial-based analyses. An article was excluded if (1) interventions did not involve digital technology; (2) vaccination was part of a multiple-aspect intervention program; (3) partial health economic evaluation; and (4) studies were presented in a format other than journal articles (such as letters, editorials, conference abstracts and posters, comments, and thesis).

EndNote was used to collect and handle all retrieved records. After deleting duplicates, screening was performed separately by 2 reviewers (WY and GF). The third reviewer (JHSY) deliberated to resolve disagreements (if any) between reviewers (WY and GF). Titles and abstracts were evaluated for eligibility in the first round. In the second phase, full-text reviews were conducted to verify potentially eligible studies based on inclusion and exclusion criteria. The whole selection method adhered to the PRISMA statement on preferred reporting items for systematic reviews and meta-analyses [18,19].

Two reviewers (WY and GF) worked independently to extract data using a standardized data extraction checklist. Disagreements during data extraction were discussed to reach a consensus. The data systematically extracted from each included study were (1) general study information (title, author, publication year, and country); (2) disease and vaccine information (disease, vaccine, and population); (3) study methodology (type of cost-effectiveness analysis, type of study design, study perspective, time horizon, digital intervention, and sensitivity analysis methods); (4) model characteristics (if applicable for model-based analyses); (5) trial or study characteristics (if applicable for trial- or study-based analyses); and (6) cost-effectiveness analysis results (incremental cost and health gains, incremental quality-adjusted life years [QALYs], incremental cost-effectiveness ratios [ICERs], and influential parameters identified by sensitivity analyses).

The Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 checklist was used to assess the methodological completeness of all studies included in this review [20]. The CHEERS checklist contains a total of 28 items on six categories: (1) title, (2) abstract, (3) methods, (4) results, (5) discussion, and (6) other relevant information. The following grading method, used for the quality assessment in the previous systematic reviews [21,22], was adopted in the present assessment: 1 point was granted to each CHEERS item when the item-specific requirements were fully complete, 0.5 point for partially satisfied, and 0 point for not satisfied. Each included study was then categorized into 1 of 4 completeness categories: excellent (scored ≥85%), very good (scored between 70% and 84%), good (scored between 55% and 69%), and insufficient (scored <55%) [21,23]. All studies were scored independently by 2 investigators (WY and GF), and disagreement was resolved by discussion with a third investigator (JHSY).

A flowchart was used to illustrate the screening and selection process, including the number of included and excluded studies. The descriptive data and significant cost-effectiveness findings of the included studies were summarized. The DHI for improving vaccine uptake was considered cost-effective if it was either (1) more effective and less costly than the comparator or (2) more effective at a higher cost and the ICER was lower than the willingness-to-pay (WTP) threshold. The completeness assessment of each included study was reported.

The systematic review identified 6860 records in target databases and 19 records by manual searches. After the removal of duplicates, 4357 studies remained. Studies were screened by title and abstracts, leading to 44 full-text articles for eligibility. Finally, 7 studies were included in the systematic review, and the flow of the whole selection is illustrated in Figure 1.

The characteristics of all included studies are shown in Table 1. Four studies (4/7, 57%) were model-based cost-effectiveness analyses [24-27], and 3 (3/7, 43%) were trial-based studies [28-30]. Three of 4 model-based studies (3/4, 75%) used the decision tree model [24-26], and 1 (1/4, 25%) used the dynamic transmission model [27]. Six studies reported the cost per unit of improved effectiveness (expressed as return case for immunization) [24-26,28-30] and 1 study reported the additional cost per QALY gained [27]. HPV vaccine [27], influenza vaccination [24], measles-mumps-rubella (MMR) vaccine [25,30], and vaccines for children at different ages [26,28,29] were the targeted vaccines. The DHIs delivery platforms were television-delivered education campaign [24], web-based decision aid [30], reminder by SMS text messaging [29], telephone (n=3) [26-28], and computer-generated recall letter [25]. Six of the 7 (6/7, 86%) studies targeted at children vaccination [25-30], and 1 study focused on elder individuals [24]. Three studies were conducted before 2000 [25,26,28], and the other 4 studies were conducted between 2014 and 2020 [24,27,29,30]. Most of the studies (6/7, 86%) were conducted in high-income countries including the United States [24-28] (n=5) and the United Kingdom [30] (n=1), and 1 (1/7, 14%) was conducted in a low- and middle-income country (Nigeria) [29].

The cost-effectiveness analyses were conducted from the perspective of society [24,28], government’s perspective [27,29], perspective of payer [25,26], and 1 study applied the perspective of the National Health Service (NHS) and society [30]. The dynamic transmission model used a long-term horizon (50 years) [27], whereas the other 6 studies used short-term timeframe ranging from 4 months to 1 year [24-26,28-30]. All studies have performed sensitivity analyses using probabilistic sensitivity analyses [24,27,30], 1-way sensitivity analysis [24], 2-way sensitivity analysis [25], and scenario analyses [25-29]. Except for 1 study with no funding [24], the included studies were all funded by national public health organizations (the National Institutes of Health [27], the Centers for Disease Control and Prevention [25,26,28], the National Institute for Health Research [30], and Japan International Cooperation Agency [29]).

The CHEERS 2022 checklist items and detailed quality assessment for each included study are listed in Multimedia Appendix 2. There were 0 (0%), 5 (71.4%), 2 (28.6%), and 0 (0%) studies classified as excellent, very good, good, and insufficient, respectively. The mean CHEERS score was 19.7 (SD 2.16, range 16.5-23; full score 28). Overall, 12 items were fulfilled by all studies (100%): (1) background and objectives, (2) health economic analysis plan, (3) study population, (4) setting and location, (5) comparators, (6) selection of outcomes, (7) measurement of outcomes, (8) rationale and description of model, (9) analytics and assumptions, (10) summary of main results, (11) study findings, limitations, generalizability, and current knowledge, and (12) source of funding. There were 11 items fulfilled in ≥50% of the studies: (1) title, (2) abstract, (3) perspective, (4) time horizon, (5) discount rate, (6) valuation of outcomes, (7) measurement and valuation of resources and costs, (8) currency, price date, and conversion, (9) characterizing uncertainty, (10) study parameters, and (11) effect of uncertainty. The remaining 5 items were only fulfilled in <50% of the studies: (1) characterizing heterogeneity, (2) characterizing distributional effects, (3) approach to engagement with patients and others affected by the study, (4) effect of engagement with patients and others affected by the study, and (5) conflicts of interest.

This systematic review included the full-scale cost-effectiveness analyses of DHIs for improving vaccination uptake and identified a small but growing number of evidence-based studies (n=7). All the included studies were found to show high level of quality (ranging good to very good) per the CHEERS checklist. The 7 included studies were published over a long time span from 1997 to 2020, and 5 (5/7, 71%) have been published since 2000 [24,27-30]. The DHIs were delivered through automated telephone calls to targeted patients in the 2 studies before 2000 [25,26]. There was increasing application of different digital technologies for improving vaccine uptake, and the cost-effectiveness was examined in the studies reported since 2000. Most studies were conducted in resource-rich countries, but one was reported in low- and middle-income countries recently in 2020 [29]. Studies with financial support were all funded by public organizations, demonstrating that the public sector has a strong interest in implementing cost-effective DHIs to improve vaccination uptake.

All studies agreed that digital technology has a cost-effective impact on vaccination uptake. One study concluded that the web-based decision aid group was cost-saving compared to the leaflet group and the usual practice group [30]. Another study, focusing on centralized reminder and recall, QI visits, and school-located vaccination, reported that the 3 interventions were all cost-effective per QALY gained [27]. The other 5 studies used cost per additional vaccinated and/or return case as outcome measures, and the results ranged from US $2.14 to US $17.79 when compared with no intervention [24-26,28,29]. However, there was no standard WTP threshold to draw a conclusion when cost per additional vaccinated or return case was used as the outcome measure. It could only be judged whether the results were within the acceptable range by comparing them with other similar studies or using the cost-effectiveness acceptability curve to express the probability of each intervention under different WTP thresholds.

The cost-effectiveness of a DHI is highly impacted by the technology cost and the change in resource usage caused by the clinical effect of technology. The intervention-delivering cost was found to be the main cost driver in all studies. Notably, the cost of DHIs was identified as the cost-effectiveness influential factor in 3 studies [24,26,28]. The included studies contained 4 model-based studies and 3 trial-based studies. For trial-based studies, the effectiveness outcomes were sourced from 1 trial with low heterogeneity, and the cost-effectiveness findings, therefore, were subject to trial-related uncertainties. All trial-based studies had conducted a sensitivity analysis to examine the robustness of results upon variation of key parameters. For model-based studies, the effectiveness parameters were sourced from multiple trials with heterogeneity, and the impact of model input uncertainties on cost-effectiveness findings was also examined in sensitivity analyses.

Our review identified some research gaps in the health economic evaluations of DHIs for improving vaccination uptake. First, cost-effectiveness research focused on social media platforms, such as Facebook, Twitter, or YouTube, is scarce. Over the last 2 decades, the advancement of technology on the web and mobile devices has fueled the development and popularity of social media, which has highly integrated into daily lives and is ready to influence the choice preferences of the population. The feasibility and impact of Facebook in promoting vaccination uptake were recently examined in clinical trials [34,35]. The cost-effectiveness of using social media platform as a DHI to improve vaccination uptake is therefore highly warranted. Second, the included studies have primarily focused on a few vaccines (HPV, MMR, and influenza) or age-specific cohort (children immunization). Evaluation of DHIs on the uptake of other vaccines such as the COVID-19 vaccine, hepatitis vaccine, and varicella-zoster vaccine is also in demand. Third, most included studies used a relatively short time horizon (approximately 1 year), limited to the follow-up period in randomized clinical trials. Cost-effectiveness studies with a long-term time horizon are lacking to evaluate life-ling outcomes of the impact of DHI for improving vaccination uptake. Finally, the cost-effectiveness analysis conducted in middle-income or Asian countries is lacking. In these countries, smart device subscription continues to rise due to web-based penetration, which potentially promotes the development of mobile health. The health care system begins to use DHIs for a wide spectrum of disease management, from prevention to treatment, in transitional countries [36]. The effect of DHI for promoting vaccination programs has been explored in Asia countries. A randomized controlled trial evaluated the effect of messaging education video to pregnant women in China. The findings showed that video education was significantly associated with a 4.8-fold increase in varicella vaccination coverage among their newborns [37]. Educational audio capsules and voice reminders delivered by mobile phone for improving child immunization coverage were assessed in a cluster-randomized pilot trial in India. The results found that mobile phone access was one of the key determinants of mobile health uptake [38]. Cost was also identified as a primary barrier to using mobile health [36]. Health economic evidence is warranted to evaluate the impact of DHIs on vaccine uptake, and essential to inform policy makers on the acceptable cost-effective strategies for applying DHIs in middle-income or low-income countries for vaccination programs.

The present review included a comprehensive search of databases and sources, and the PRISMA checklist was observed during the process. There were yet some limitations in the present systematic review. The search strategy of the systematic review only included studies written in English, and some relevant studies (in non-English languages) may be missed. The included studies applied different effectiveness outcomes and therefore limited the comparison of cost-effectiveness findings between studies. Also, the included studies were conducted in countries with different health care systems and cultures. The generalizability of health economic evidence should be cautious to scenarios of similar contexts, including, but not limited to, vaccination type, target individuals, health care systems, accessibility of mobile network, and culture.

This is the first systematic review on cost-effectiveness analyses of digital technologies for improving uptake of vaccination. The included 7 cost-effectiveness studies showed “good” to “very good” quality per the CHEERS checklist. All included studies reported that the DHIs were cost-saving or cost-effective for improving vaccination uptake. The paucity of cost-effectiveness studies on DHIs for vaccination uptake indicated that further long-term evaluations on social media-delivered vaccine-specific DHIs are highly warranted.