We conducted a systematic review of systematic reviews—an approach that has proven helpful in synthesizing a broad base of literature in order to identify research gaps and inform future intervention programs [14]. Our aims were to assess the evidence of effectiveness of asthma digital interventions for self-management as measured by an inclusive range of clinical and process outcomes.

We documented recruitment and retention rates, information about implementation processes, whether cost effectiveness was assessed, and whether theories of behavior change were used in intervention development to help us gain a better of understanding of features that helped or hindered the use of the programs. Our protocol is available in Multimedia Appendix 1.

We included qualitative and quantitative reviews and used the PICOS (participants, interventions, comparison, outcomes, study design) framework [15] to define inclusion criteria (see Textbox 1 for details). Our exclusion criteria can be found in Textbox 2.

A professional systematic review company (York Health Economic Consortium) searched a wide range of databases covering health, mental health, education, social science (14 in total), with no start date before July 2011. The search strategy was devised using a combination of subject indexing terms (eg, MeSH [medical subject headings] in MEDLINE), and free-text search terms in the title and abstract. The search terms were identified through discussion between the research team, by scanning background literature, and by browsing a database’s thesaurus. To ensure sensitivity, the search strategy did not include a methodological search filter to identify reviews. The searches were not limited by date range or language. Hand-searching of Patient Education and Counseling, and the Primary Care Respiratory Journal, and reference searching and citation searching of included studies was undertaken (DM). We contacted experts to establish if any reviews had been missed. The search of electronic databases was updated to October 3, 2013.

The search strategy covered 3 broad areas: (1) asthma and related terms, (2) online/computerized and related terms, and (3) self-care/self-management, patient experience, qualitative, and related terms. The full list of databases searched and an example of the full search strategy for MEDLINE are available in Multimedia Appendix 2.

Titles, abstracts, and full papers were screened by one researcher (DM) plus one other independent researcher (EC, SW, FM, NCT, KA, RD, AM, or VR). We independently undertook quality appraisal; disagreements were resolved by discussion with a third party if necessary. Only studies meeting predetermined quality criteria advanced to data extraction.

We used online data collection forms using Distiller SR software. For each included review, we collected (1) general information about the review (year, country of first author, language, number of studies, number of asthma studies, and number of digital asthma studies), (2) descriptions of each included RCT and intervention that fulfilled our criteria (inclusion criteria were applied to the systematic reviews initially and then subsequently to their featured RCTs, to determine those relevant to this review), and (3) results for each outcome of interest and the original article this result was derived from (including quotes from qualitative/narrative reviews).

Quality appraisal was undertaken in two ways. First, at the full paper screening stage, papers were required to meet criteria laid out in our definition of a review (eg, evidence of a systematic search or criteria for selection of papers must be included; see Multimedia Appendix 1 for full definition). Then, the included papers underwent formal quality appraisal using A Measurement Tool to Assess Systematic Reviews (AMSTAR) [16-18]. This 11-point checklist covers the following domains: establishing the research question and inclusion criteria before the conduct of the review, data extraction by at least 2 independent data extractors, comprehensive literature review with searching of at least two databases, detailed list of included/excluded studies, quality assessment of included studies and consideration of quality assessments in analysis and conclusions, appropriate assessment of homogeneity, assessment of publication bias, and a statement of any conflict of interest. We made minor alterations to the wording in order to make the checklist applicable to non-quantitative reviews (available on request from the corresponding author). Papers needed to achieve at least 50% of quality indicators, plus a “yes” to question 7, which asks “Was the scientific quality of the included studies assessed and documented?” This was required to allow us to make a comment on the quality of the included data. Two researchers (DM + [KA, RD, AM, or VR]) scored each paper independently, with conflicts resolved by discussion with a third party if necessary (FM).

Any data provided describing risk of bias across reviews, and within individual RCTs, were extracted to inform the discussion.

Meta-analysis was not possible. Quantitative results for individual outcomes results were described as favoring the intervention group, favoring the control group, or demonstrating no difference. All results were included in a narrative summary.

Our search identified 3810 unique citations, and title and abstract screening identified 116 full papers for review. Of these, 29 fulfilled our inclusion criteria (Figure 1).

The 29 papers included systematic reviews from United States (12), United Kingdom (6), Canada (3), Portugal (2) Australia, France, Spain, Norway, Taiwan, and Greece (1 each). Only 9 reviews focused completely on asthma interventions (the others included self-management interventions for asthma alongside a range of other chronic conditions such as diabetes, heart failure, and hypertension), and 11 reviews looked exclusively at digital interventions (the rest examined digital interventions alongside other modes of self-management). These 29 reviews consisted of 4 Cochrane reviews [19-22] and 3 meta-analyses [23-25], with the remaining 22 being narrative or descriptive studies (see Table 1; [10,19-47]).

Within the 29 included reviews, descriptions of the included RCTs and particularly the interventions themselves were generally suboptimal. AMSTAR scores ranged from 10% to 100% with a median of 40% (interquartile range 20–50), with 10 reviews scoring 50% or greater. All reviews with 50% or over also provided an individual assessment of quality, which was an essential criterion to progress to data extraction. Areas where reviews performed particularly poorly include providing information about conflict of interests of included RCTs (1/29), providing a priori design (6/29), independent screening and data extraction (6/29 clearly demonstrated this; a further 15/29 may have, but insufficient information was provided to be able to confirm this), and providing a list of excluded studies (7/29).

Of the 10 reviews meeting the predetermined AMSTAR requirements (Table 1), these included the 4 Cochrane reviews and 6 narrative/descriptive reviews. Four reviews focused only on asthma, with the remaining 6 featuring a range of conditions. Four of the reviews included nondigital/electronic modes of delivery of interventions. There was no one review that looked specifically at digital interactive interventions aimed only at those with asthma. These 10 systematic reviews presented results from a total of 19 RCTs meeting our inclusion criteria (see Table 2 for descriptions [10,19-22,34,37,42-44,47-68] and Table 3 for interventions [48-66]).

All 10 systematic reviews provided sample sizes for their included RCTs, but only three provided numbers allocated to intervention group versus control [19,20,37]. All reviews presented some information about age, 8 out of 10 presented information about gender, and only 4 of the 10 reviews about the ethnicity of participants [19-21,44]. No review provided information about socioeconomic status or levels of educational attainment.

Seven of the reviews provided risk of bias data based on the guidance provided by the Cochrane collaboration [69], whereby different elements such as adequate sequence generation, allocation concealment, and blinding are assessed as being of low risk of bias, high risk of bias, or unclear risk of bias. This does not provide an overall score. Two reviews provided quality scores, one based on the Oxford Quality scoring system [47] and the other using the Consort Statement [34]. The final review [10] provided an overall quality grade based on guidance from within the Cochrane handbook, but did not provide any rationale for their grading. Grading of RCTs in this way is subjective, and there was conflict between reviews about the risk of bias present in a given RCT. For example, the presence of adequate sequence generation by Guendelman was assessed by 4 different reviews [19,22,42-44]. Two described this as “unclear risk of bias”, but the other two reported it as “adequate”, and “low risk of bias”. This, combined with the various different methods used and in some cases limited information provided, meant that we are unable to make any detailed statements about the quality of the included trials.

The 19 unique RCTs within the reviews are described in Table 2. Reporting of descriptive data about RCT participants was mixed; only 4/10 reviews [19-21,44] provided information for all 7 descriptive headings (sample size, age of participants, gender, ethnicity, dropout rate, duration, country of study), with no data about socioeconomic status. The sample sizes for the 18 RCTs providing quantitative results ranged from 20 to 378. There were discrepancies in the sample sizes reported for some RCTs between reviews; for example, the RCT by Jan had 3 different sample sizes reported from 4 reviews (164, 179, and 196). Where there were discrepancies, taking the largest number provided the total number of participants of 2315. From the 19 RCTs featured, 4 were aimed at adults, 14 at children or adolescents, and the age range for one was not described (HussA). All 4 of the RCTs aimed at adults had upper age limits (of 25, 45, 50, and 65 years).

From the 8 RCTs with information about gender, 9 had a majority of female participants and 9 had a majority of males; combining the RCTs with sample size and gender numbers, the percentage of male participants was 54% (793/1478). Dropout rates were available for 11 of the 19 RCTs and ranged from 4.5% to 23%, but again, the reporting of these numbers for 3 of these RCTs was conflicting (Krishna, Jan, Guendelman). Reasons for dropout were rarely provided. Duration of studies ranged from a one-off use of the intervention, to access for 12 months. Eleven RCTs were from the United States, and one each from Taiwan, Portugal, Sweden, Denmark, and the Netherlands. The country was not described for the remaining 3 RCTs.

A summary of the key components or ingredients reported as being present in the interventions is summarized in Table 3. Provision of information and self-management education was the most common feature present in 16/19 RCTs. This was followed by the presence of immediate feedback/interactivity from the device (13/19).

Ten of the interventions used games/quizzes/vignettes, and all of these were from RCTs from 2006 or earlier. Eight of the interventions involved some form of direct communication either to or from health professionals. Six of the interventions were available for daily use. The presence of an action plan was noted in 4 interventions, all 2005 or later. This was the same for interventions featuring automated reminders/alerts from devices. Description of the interventions was variable between reviews, and it is possible that many of these interventions feature components not described in Table 3.

The key ingredients or components of interventions were often poorly described rendering it impossible to draw conclusions about the effects of different components of interventions on outcomes.

There were quantitative results available for the following outcomes of interest: activity limitation, knowledge, markers of self-care, quality of life, medication use, symptoms, missing school, lung function, and health service utilization (eg, emergency department visits, hospitalizations, primary care visits).

There were descriptive results relating to adverse events, behavior change theory use, and patient satisfaction. No data were provided about cost-effectiveness or biomarkers of airway inflammation.

Quantitative results were provided in a range of ways across the reviews. For example, McLean et al [22] provided original numbers of events and sample size, with results given as odds ratios with confidence intervals, whereas Stinson et al [34] reported the outcome as either statistically significantly in favor of the intervention group (+), the control group (-), or no difference (0). Murray et al [19] presented results as either standardized mean differences (SMD) and effect size (in the form of Lipsey Categories of small, medium, or large) or odds ratios. However, the significance for individual RCT results was not provided, limiting the conclusions we can draw from this individual review (therefore not included in the text below). Quantitative results are presented in Table 4 and Multimedia Appendix 3 and summarized below. Multimedia Appendix 3 details all the data extracted from the reviews, along with a summary stating whether the results favored the intervention group (Y), showed no difference (0), or statistical significance of results was not provided (n/a). No results favored the control groups. Table 4 summarizes only the results that provide a measure of statistical significance [10,20-22,34,37,42-44,47].

Stinson [34], Johansen et al [42,43], and Kirk [44] reported findings from two RCTs (Joseph and Guendelman). Both reported that the use of a digital intervention reduced the number of days of restricted activity significantly.

Three reviews [34,42-44] provided results from 2 RCTs (Jan, Krishna) about the impact of online interventions on knowledge, both of which provided results in favor of the use of digital interventions.

Six of the reviews [10,34,37,42-44,47] presented data for markers of self-management from 7 separate RCTs (Jan, Joseph, Rubin, Vilozni, Guendelman, HussA & Rasmussen) covering, for example, proportion using action plans, spirometry/inhaler technique, diary adherence, and impact on completion of self-management sessions. All except one (Guendelman) showed a positive effect.

Six reviews [21,22,34,42-44,47] presented data from 7 RCTs for quality of life. Four of the RCTs favored the intervention group (Jan, Van der Meer, Shames, Rasmussen), and 3 showed no difference (Joseph, Krishna, Kamps).

Two reviews [37,47] provided results from 3 RCTs for this outcome. Two favored the interventions group (McPherson & Huss), while the other showed no difference (Shames).

Seven systematic reviews [21,22,34,37,42-44,47] provided results for this outcome, from 12 RCTs. Six reported no difference (Guendelman, HussA, HussB, Shames, Kamps, Sundberg); the remaining 6 studies favored the use of the digital interventions (Jan, Joseph, Krishna, Rasmussen, Van der Meer, Bartholomew).

Three reviews reported for this outcome [22,34,47] from 4 RCTS: two favoring the intervention group (Joseph & McPherson), and two showing no difference (Guendelman & Rubin).

Six reviews [20-22,37,42-44] presented data for this outcome from 7 RCTs, with three favoring the use of digital interventions (Rasmussen, Guendelman, Sundberg), and 4 showing no difference (Jan, Shames, Kamps, HussA).

Five reviews [22,34,42-44,47] provided results from 6 RCTs on emergency department (ED) visits. Four trials showed no difference (Rasmussen, Bartholomew, Jan, Guendelman), and 2 studies favored the intervention (Joseph, Krishna).

Five reviews [22,34,42-44,47] provided information from 5 RCTs, all of which showed no significant differences between the intervention group and controls (Guendelman, Joseph, Rubin, McPherson, Rasmussen).

Two reviews provided data for this outcome [44,47] from 4 RCTs (Shames, Rubin, McPherson & Jan). All 4 RCTs demonstrated no difference in the number of visits.

Initial full paper screening resulted in 29 systematic reviews. However, following quality appraisal, only 10 met the predetermined AMSTAR cut-off values. This metareview summarizes the findings from these 10 systematic reviews featuring a total of 18 unique RCTs of digital asthma self-management interventions. Only 11 of the 19 RCTs were present in more than one review suggesting that at present there is no single good quality systematic review looking specifically at digital self-management interventions for asthma.

Within the systematic reviews we found that information about methods, intervention components, and results were often brief and did not allow for meaningful comparisons between interventions. The described studies themselves were extremely heterogeneous making comparison between interventions difficult. “Control” groups ranged from no active intervention or contact with health professionals to multiple face-to-face teaching sessions and intermittent use of the intervention itself, which may have masked potential positive outcomes of digital interventions. More recent reviews tended to have more information, particularly with online appendices.

In no studies did the control groups have better outcomes, although only two reviews addressed the issue of adverse events, and specific information was available from only one study, which is a concern. That study suggested digital interventions groups may be at higher risk of adverse events related to the fact that successful interventions usually increase inhaled corticosteroid use (a positive outcome) and therefore results in more cases of dysphonia or oral candidiasis. Surprisingly, the issue of patient satisfaction was also neglected, being specifically addressed in only one trial, which suggested participants preferred a Web-based system to a paper-based one, and discussed in general terms in one further review.

Descriptions of intervention development and particularly the use of theory (which has been shown to increase effectiveness) were either brief or not discussed at all, although our review suggests that more recent reviews are increasingly recognizing this, an issue highlighted by the recent publication of a CONSORT EHEALTH statement [70]. This matters because trying to establish the key components of effective self-management interventions is a challenge faced by researchers in this area [71]. For example, a recent Cochrane review of computer-based interventions in adults with type 2 diabetes concluded that there was a small beneficial effect on blood glucose control, particularly in the mobile phone subgroup [72]. However, the authors also commented that the key “ingredient” of effective interventions was unclear; it could either have been that the mobile phone itself was important or that the interventions delivered using mobile phones had included BCTs that were likely to lead to success, and these could be the “ingredient” that made these interventions more effective. What is consistent across several systematic reviews on digital support for self-management of chronic illness is that interventions with multiple behavior change techniques appear, on the whole, to be more effective than those using fewer and that the use of theory to inform the choice and combination of BCTs appears to be associated with increasing effectiveness [72-74].

Only four RCTs featured adults, and there were no participants over the age of 65 years, with only one small trial with just 21 participants including individuals over the age of 50 years. No information was provided about the socioeconomic status or education level of participants. None of the trials were undertaken in low-income countries. Attrition rates of up to 23% were recorded. This is in keeping with other systematic reviews of digital interventions in other areas. For example, a recent Cochrane review examining computer-based weight loss interventions found attrition rates ranging from 2-25% (median 16%) [75], and attrition may be worse in interventions targeting older age groups, with one review including digital and nondigital interventions noting rates between 0 and 52% (median 15%) [73]. Reassuringly, attrition rates are no worse than those found with nondigital self-management asthma interventions as described in Gibson’s Cochrane review examining asthma self-management education and regular health professional review, where attrition rates ranged from 0-54% (median 15%) [8].

Our metareview suggests that digital interventions may be effective at improving knowledge, reducing activity limitation, improving markers of self-management, improving quality of life, and optimizing medication use in those less than 65 years of age. However, certain indicators, such as knowledge and activity limitation, were assessed in only two trials, and medication use improvements noted in two out of three trials that examined this. There was no evidence of improvements in symptoms, lung function, school absences, or health service utilization.

Importantly, we found no qualitative synthesis of asthma digital interventions that would have given insight into the patient experience of using digital self-management interventions, implementation processes, or barriers and facilitators to their use, which is an important gap in the evidence base.

Our metareview has a number of strengths and limitations. The search was undertaken by a team with good experience of systematic reviews, using multiple databases, and using a strategy designed iteratively with researchers to be as inclusive as possible, without being unwieldy. Despite this, we may have missed reviews of chronic illness interventions including asthma but not specifically indexed as such. We included non-English studies, which is a strength. Due to the heterogeneity of the data, a formal metasynthesis was not possible.

A further limitation here is the reliability and comprehensibility of the included information. When undertaking a review of reviews, the data are an extra step away from the original research increasing the possibility of reporting errors. We noted several discrepancies between reviews describing the same RCTs, in sample sizes and gender descriptions. Some trials presented data as sample size, while others used terms such as number of participants, and dropout calculations did not appear to correlate on several occasions. This could be due to reporting error, or differences in the interpretation of the terms sample size, follow-up, and participants. In addition, there was a lack of detail describing the populations included, and limited definitions of outcome measures provided by featured systematic reviews. This translates into a lack of specific detail about the population our results are relevant to, and limitations in comparing results that have potentially used different ways of defining outcomes. Finally, establishing a minimum standard of quality of included reviews ensures that there is a degree of reliability to the conclusions [76]; however, this undoubtedly narrowed the available data from which we could draw conclusions. Had we not used AMSTAR criteria, we could have had data from up to a further 12 RCTs and two qualitative studies. However many of these RCTs were included only in low AMSTAR scoring papers (ie, those that had not undertaken any quality appraisal of their included studies), and there were often no descriptions of control groups, or more than a few words describing the interventions, and therefore inclusion of any such data would have rendered our conclusions meaningless. The RCTs themselves often appeared poor quality (eg, poor randomization strategies), with small numbers (eg, sample sizes as low as 10), and had been excluded from the more robust reviews for these reasons. The lack of economic data is a weakness, although the results on health care resource use (hospitalizations and ED visits) suggests that evidence of cost-effectiveness may be lacking. However, without data including routine health care resource utilization and formal economic analysis, no firm conclusions can be drawn.

This metareview provides a snapshot of current knowledge about effectiveness of digital self-management support for those with asthma. Digital self-management interventions show promise, with evidence of beneficial effects on some outcomes. However, we know nothing about the socioeconomic status of participants, and few over the age of 50 years and no one over 65 years of age were included. Thus, the true “reach” of these studies is uncertain, and their likely uptake and use by the wider population of those with asthma remains uncertain. Few interventions were underpinned by robust theoretical frameworks. Digital interventions are poorly described, and there is insufficient information about barriers and facilitators to their uptake and utilization. Importantly, patient perspectives have been largely ignored in currently available reviews. There was little data about cost effectiveness within reviews, but this appears to relate to the lack of existence of such data from primary trials. Digital interventions for asthma appear promising but further robust investigation is needed, first, in the form of a detailed systematic review of currently available digital interventions aimed at those with asthma, detailing the presence or absence of BCTs. Second, examination of the primary qualitative literature to describe what is already known about the patient’s perspective would be invaluable to inform future interventions.