• Home
• About
• Log In
• Register
• Search
• Upcoming
• Current
• Archives
• Announcements
• Subscribe
• Submit
• Blogs

	Members can download this full article for Adobe PDF Format. Membership provides unlimited access to all PDF files. Or, ask your department head to become an institutional member.
	Buy Now:
	Price (USD): $9.00

Review

The Effectiveness of Web-Based vs. Non-Web-Based Interventions: A Meta-Analysis of Behavioral Change Outcomes

Dean J Wantland¹, MS, RN, MSN; Carmen J Portillo¹, PhD, RN, FAAN; William L Holzemer¹, PhD, RN, FAAN; Rob Slaughter², PhD; Eva M McGhee^1,3, PhD

¹Department of Community Health Systems, School of Nursing, University of California, San Francisco CA, USA
²School of Nursing, University of California, San Francisco CA, USA
³UCSF Comprehensive Cancer Center, School of Medicine, University of California, San Francisco CA, USA

ABSTRACT

(J Med Internet Res 2004;6(4):e40)
doi:10.2196/jmir.6.4.e40

KEYWORDS

Web-based intervention; non-Web-based intervention; Web-based therapy, Internet; meta-analysis; patient outcomes; adults

A primary focus of self-care and self-management interventions is the encouragement of an individual's behavior change in the presence of a chronic illness or condition necessitating knowledge sharing, education, and understanding of the condition. There has been limited research comparing the use and effectiveness of Web-based interventions to non-Web-based interventions such as traditional face-to-face interactions and paper and pencil assessments. The introduction of the Internet into clinical practice as an information-sharing medium has brought about many opportunities for innovative interventions for individuals with chronic illnesses and their care providers. These interventions are often designed to address deficiencies in patient knowledge and chronic illness self-management skills. Improvements in these areas have been shown to lead to improved health outcomes. However, the extent of the benefits gained through the implementation of Web-based self-regulatory and behavior change interventions compared to non-Web-based interventions has not been fully ascertained. This meta-analysis was undertaken to establish any potential effect size differences between Web-based and non-Web-based interventions on selected patient behavior change outcomes.

In recent years, there has been an increase in the use of the Internet to gather, transform, and disseminate information that, in earlier years, was primarily done through the use of paper, in the form of books, pamphlets, instruction materials and so on. Internet users are seeking health information and healthcare services; 80%, or about 93 million Americans have searched for at least one of 16 major health topics online [1]. The Robert Wood Johnson Foundation (RWJF) has noted the increased use of Internet-based devices, cellular phones, and personal digital assistants (PDAs) creating opportunities for both patients and providers to benefit from access to e-Health applications. The RWJF has supported this trend by providing funding to study health behavior modification and chronic disease management in nontraditional settings through the use of e-Health technologies [2]. The use of computers to directly collect health assessment data from patients is a well-established technology that has been shown to produce reliable responses when administered over the World Wide Web [3]. In some circumstances, computer surveys have been shown to have advantages over face-to-face interviews. In one study, computer-based screening elicited more HIV-related factors in the health histories of blood donors than did standard questionnaire and interviewing methods [4]. Participant disclosure of high-risk sexual encounters has also been improved given the semblance of the more anonymous, Web-based data collection methodologies [5].

Computerized health behavior interventions are beneficial to patients/clients and healthcare providers. This is evidenced by structured reviews on the effectiveness devices such as kiosk-based computer assisted self-interviewing, interactive video, Internet applications, computer aided instruction, and the like in a variety of patient care settings. Balas and colleagues found that interactive patient instruction, education, and therapeutic programs helped individuals improve their health; at the same time, healthcare delivery processes were also improved [6]. Research studies suggest that education and knowledge sharing benefits can be achieved through computer-based education methodologies [6,7].

[view this figure]

Figure 1. Search terms “Web-based Therapy” trended by year of publication

Interest in use of the Internet and Web-based interventions is increasing rapidly. In the 7-year period from 1996 to 2003, a total of 569 citations demonstrated a twelve-fold increase in MEDLINE publication citations for “Web-based therapies,” from 13 citations in 1996 to 152 citations in 2002. There has also been a steady increase in the number of citations in MEDLINE for the term “Web-based intervention,” further indicating interest in this research area for Web-based treatments. In addition to completed patient-focused, Web-based intervention studies, a large number of the publications are simply proposed or newly implemented studies. Many studies are based on therapeutic interventions that are provider focused and part of an implemented system incorporating the use of computerized medical records. Others include telehealth technologies that include highly technically interfaced lab values recorded within a case managed setting. Others discuss the variety and integrity of health-related Web sites (Figure 1).

Data Sources/Systematic Review

For identification of the relevant literature, a specific search strategy was performed using explicit inclusion criteria to avoid selection bias. A MEDLINE, CINAHL, EMBASE, ERIC, and PSYCHInfo search between the years 1996 and 2003 was conducted using keyword search terms of “computerized intervention,” “Internet intervention,” “Web-based therapy,” and “Web-based intervention.” The Cochrane Library collection was also accessed using keyword searches for “Web-based intervention” and “Internet intervention.” Searches in additional databases were done but revealed no new comparative Web-based published articles. A manual review of the reference lists of these articles was done to identify additional articles for possible inclusion. When an article was identified, it was compared against established inclusion/exclusion criteria to determine its suitability for the meta-analysis. The inclusion/exclusion criteria are presented in Table 1.

[view this table]

Table 1. Inclusion and exclusion criteria for the meta-analysis

Quality Documentation of the Studies

The quality assessment of the included studies was based on the method used by Haynes and colleagues [8], with modifications to address the focus of this study on Web-based interventions. The compliance to standards for the studies is based on five criteria: (1) study design; (2) selection and specification of the study sample; (3) specification of the illness/condition; (4) reproducibility of the study; and (5) outcomes specification and the measurement instruments used/validity and reliability documentation of instruments. The sum of the variables result in a total score ranging from 0 to 18 (Table 2). Only studies with a quality documentation score of 12 or greater were retained for the meta-analysis.

[view this table]

Table 2. Quality evaluation of selected investigations (adapted from Haynes et al [8])

Instrument Reliability and Validity

It is important to compare Web-based study instruments to their counterpart paper-based study instruments. Structured assessment instruments can be used to reliably measure a broad range of attributes of patient health and status. For comparative purposes in a meta-analysis, it is important to know the reliability of the measurement instruments with the reliability of the item measures reported in the publication. The validity and reliability of a Web-based measurement approach itself has not yet been adequately addressed. It cannot be assumed that the validity of an instrument based on its paper format and use in a specific research situation is transferable to the instrument's use in a Web-based format. Some instruments may be modified in ways that could change their meaning and accuracy, such that it might be inappropriate to compare data collected from different versions of the instruments (for example, provider administered assessments vs. self assessment). The ordering of the questions within an instrument can affect reliability and validity. In a Web-based format, the expected ordering may change and the ability to go back and review/change answers may need to be considered. The format of text can affect how the questions and instructions are interpreted. The use of bolding, italics, colors, fonts, and capitalization can affect the readability of items and change their phrasing. These can also draw attention to or from key parts of the instructions [9].

Effect Size Calculation

A number of studies have been conducted having a measure that can be compared for its effect size in both a Web-based intervention vs. a non-Web-based intervention. Although the studies vary in the use of different outcomes that are used as measures for knowledge and/or behavior change, the construct of such change may be validly measured using meta-analytic techniques [10]. Although most studies had multiple outcomes from which to measure knowledge and/or behavior change, using several effect size calculations to represent results from each study outcome violates the rule of independence for statistical analysis, as these outcomes were obtained from the same sample of participants and were obtained in a similar setting. Multiple outcome effect sizes will also give disproportionate weight to studies with multiple groups and multiple scales compared to studies using fewer outcome measures.

Effect size was used to quantify the effectiveness of the Web-based intervention, relative to a non-Web-based comparison intervention. Effect size analysis was done to ascertain a standardized difference between the Web-based and non-Web-based groups, regardless of how the outcome was measured, by applying the appropriate meta-analytic technique. This analysis makes the assumption that individual studies are estimating different treatment effects and will observe the resulting effect size values and confidence intervals for distribution and variability. This check is done to evaluate if the effects found in the individual studies are similar enough that the combined effect size estimate is meaningful.

Hedges' d, a bias corrected modification of Cohen's d, was calculated to determine the magnitude of the difference between the mean of an intervention group and the mean of the control group, divided by a pooled standard deviation [10]. The calculations were based on the reported data in each of the studies that provided sample sizes, means, and standard deviations for each of the Web-based and non-Web-based intervention groups for the relevant effect (outcome) variables. A homogeneity statistic, Q_w, was also calculated to determine whether the values of d used to calculate a mean effect size were consistent within the set of the reviewed studies. Heterogeneity is indicated when the Q_w statistic has a large, statistically significant value, suggesting that one or more features that were present in some studies and absent in others were affecting the magnitude of the effect sizes.

In controlled, repeated-measures studies, the effect size was calculated using the earliest time period for controls (non-Web-based intervention) and the final time period for controls then repeated for the intervention (Web-based intervention) groups, achieving one effect size for each group. The Web-based and non-Web-based group effect sizes were integrated to achieve one effect size for each study variable reviewed. In studies where standard deviations were not reported, but P values and/or z scores were provided, the Stouffer method for effect size calculation was used [11]. In studies having frequency or proportion data, the Mantel-Haenszel-Peto method was used to calculate the effect size between the Web-based and non-Web-based intervention groups [10]. For those studies that had multiple methodologies (i.e., multiple Web-based intervention groups compared to one paper-based group) or for those studies that used multiple paper-based methodologies (i.e., self-completion of a paper assessment and provider interview), the multiple group means were combined, the standard deviations were pooled, and effect size calculated. In those studies using a case/control, repeated measures design, the calculations for effect size and analysis of the effect sizes were performed using D-Stat Version 1.0 (Lawrence Earlbaum Associates, Inc., Hillsdale, NJ). Graphing was done using SPSS version 11.5 (SPSS Inc., Chicago, IL). Drop-line charts for individual groups using the variables for effect size and the low and high confidence interval values were graphed to provide visual representation effect sizes and associated confidence intervals.

Descriptive statistics were used to ascertain means and standard deviations as needed for aggregating the study data. Participant attrition rates in the longitudinal studies were calculated from the group N at the time of enrollment into the study until the time of the final reported follow-up period.

Citation Searches

MEDLINE, CINAHL, EMBASE, PSYCHInfo, ERIC, and Cochrane Library, keyword searches resulted in 1518 citations. After reviewing for database redundancies in the citations, individual examination of the reference lists, and reviews of dissertations, a final review against the inclusion/exclusion criteria and quality documentation resulted in 20 studies selected for the instrument format analysis and the intervention-focused meta-analysis for behavior change outcomes. The selected studies were performed in the United States, France, Japan, Italy, Spain, Netherlands, Sweden, and Germany.

Exemplar studies, not selected for analysis, are summarized as follows: Studies that were Web-based to Web-based intervention comparisons [12-15]; 2) Studies that were descriptive of the functionality of a Web site [16,17]; 3) Studies that were provider focused [18]; 4) Pre/post intervention studies that only assessed the Web-based intervention [19-24]; 5) Studies that did not provide adequate information regarding either a change in outcomes or the comparative utility/validity/reliability of the Web-based tool [25-27]; and 6) Computer-assisted instruction (CAI) studies [28-30].

Characteristics of the Reviewed Studies

Review of the selected articles revealed variation in design of the Web-based intervention studies. Because of the variation in the framework for these studies, two separate analyses were performed that: (1) evaluated studies that focused on a one-time, cross-sectional survey comparison of assessment instruments/methods when administered to Web-based and non-Web-based groups [3,31-34]; and (2) evaluated outcomes variables of intervention that best indicated knowledge and/or behavior change resulting from a Web-based intervention [35-51]. A summary of each study is shown in Table 3.

Aggregation of data from the 22 selected studies showed a total of 11,754 participants in both the Web-based and non-Web-based interventions at the time of inclusion into their respective studies. Of this total, 5,841 were women and 5,729 were men. The average age of participants was 41.5 years. For longitudinal studies, the average intervention duration was 27 weeks with a range from 3 weeks to 78 weeks. Attrition rates for the longitudinal studies revealed that both the intervention and control groups lost an average of 21% of the study participants over the duration of the studied interventions. (Table 4).

[view this table]

Table 3. Summary of reviewed studies**

[view this table]

Table 4. Demographic characteristics of the cumulative studies

Knowledge and Behavioral Change Outcomes

Sixteen of the 17 studied effect outcomes revealed improved knowledge and/or improved behavioral outcomes for participants using the Web-based interventions. The individual effect sizes for each of the reviewed study variables for knowledge change and/or behavioral change showed effect sizes ranging from small (±.01 to .19) [36-38,41,44,46]; to moderate (±.20 to .47) [39,45,47,50,51]; to moderately large (.54 to .75) [40,42,43,49]. Of the 17 studied outcome variables, six showed that the positive effect sizes were statistically significant as seen by the confidence intervals being greater than zero [42-45,47,49] (Box 1). The one study favoring non-Web-based interventions did not show statistical significance [46]. There was no significant difference between the length of an intervention and effect size for the studied outcome.

Review of the forest plot graphical output figures showed a high degree of heterogeneity indicated by the confidence interval overlap (Box 1). Estimation of the homogeneity statistic was calculated and was statistically significant indicating variation between the 17 studies (Q_w16 = 49.993, P ≤ .001). Sensitivity analysis to ascertain the studies with the greatest heterogeneity, revealed three standout studies [37,46,49].

[view this box]

Textbox 1. Effect size (ES) for outcome variables in the analyzed Web-based interventions compared to paper-based interventions (N = 17 Studies)

Assessment Instrument/Methods Comparison

The five studies comparing assessment instruments/methods when administered to Web-based and non-Web-based groups revealed two studies showing moderate negative effect sizes (Wu -.24; and Soetikno -.22)[33,34] favoring the paper-based/traditional format. The remaining three instrument/method comparison studies showed small to moderate positive effect sizes ranging from .17 to .44. One of the five studies [31], showed a statistically significant effect size, indicated by zero being included in the confidence interval, the remaining four studies showed no statistically significant effect size comparison indicating little variability between the format of the instrument/method being either Web- or non-Web-based (Box 2). Analysis of homogeneity of these five studies revealed a statistically significant Q value (Q_w4 = 18.238, P ≤ .001).

[view this box]

Textbox 2. Effect size (ES) evaluation of studies assessing instruments/methods when administered to Web-based and non-Web-based groups (N = 5 Studies)

Discussion

Advantages for the Use of Web-based Interventions

The management of any chronic disease should be personalized to an individual, as the person is ultimately responsible for the success of the intervention. Self-management of a chronic condition and contribution to disease management has demonstrated improved results and adherence to treatment regimens [52]. Consequently, Web-based interventions should be designed to allow individuals to tailor the intervention to their specific needs. With the advent of high-level Web programming languages, intended to provide effective data and information provision and retrieval, the flexibility to provide interactive and responsive programs for use on the Internet is increasing. This is conducive to the incorporation of interactive and continuous self-monitoring, feedback and information exchange that is certain to play an increasingly important role for this patient care need.

Comparative Intervention Studies

Although the studies vary across many clinical areas of interest, there is a consistency of the selected outcome variables being targeted to require either or both an individual's knowledge and behavior change to achieve the outcome. The review of the individual study effect size comparisons in the use of Web-based compared to non-Web-based interventions showed an improvement in individuals using Web-based interventions to achieve behavior change for the studied outcome effect variables. The broad variability in the focus of the studied outcomes precluded the calculation of an overall effect size for the compared outcome variables in the Web-based when compared to the non-Web-based interventions. Additionally, a homogeneity statistic estimation also revealed widely differing study parameters (Q_w16 = 49.993, P ≤ .001). Sensitivity analysis ascertained three studies with the greatest heterogeneity [37,46,49], these studies were not excluded from the analysis as their contribution to the research using Web-based and non-Web-based interventions showed significant findings. There was no significant difference between study length and effect size in the longitudinal studies.

Assessment Instrument/Method Comparison Studies

A comparison of the five Web-based instruments and the non-Web-based instruments shows the variability between the formats of the instrument to be moderate to small. The effect size analysis confirms the respective authors' findings in each of their studies. For the studied instruments, the Web-based instruments produced valid and reliable results. These studies revealed effect sizes to range from -.25 to +.29, only one of which was statistically significant, favoring Web-based interventions. In the studies that measured the use of quality of life (QOL) instruments such as the MOS-HIV and the SF-36, it should be noted that in the Bell and Kahn study [3], there was no specification of any predisposing illness in the Web-based intervention group. In the non-Web-based population, the scores reported by the authors of the comparative study [53], were combined from studies with participants having varying illnesses, which may account for this comparison group having worse SF-36 scores than the anonymous comparison group. Further, these QOL instruments may not be sensitive enough to capture the illness severity of the subscales for Web-based clients. Floor effects have been reported for the SF-36 for those with severe illness related impairment [54]. Conversely, ceiling effects may be present if the Web-user is doing well and not experiencing levels of debilitation due to symptoms. The MOS-HIV and SF-36 may not possess sufficient sensitivity to change to adequately reflect the symptom experience and management of symptoms in ongoing tailored interventions requiring daily or weekly input.

Demographic Characteristics

Most of the studies explained the possibility of demographic differences (i.e., culture, age, gender, ethnicity, and/or income) in their study intervention populations. Some studies controlled for the possibility of these differences [40], while others provided training to the Web-based intervention participants [34,43,47]. In the reviewed studies, the average age of the study participants was 41.2 years, which is relatively young. It is likely that this is not the same population who are living with many chronic illnesses. Most of the studies did not discuss issues such as ethnicity, income level, or homelessness, which are important when considering the use of a Web-based technology to deliver an outpatient intervention. All but one of the studies [45] did report gender, but overall, the differences between participation of men and women were not large in the studies. Two studies looked at HIV interventions and had a preponderance of men (N = 237) with an average age of 37.5 years [34,40]. The studies by Bell et al and Christensen et al [3,37] were open access Web sites and had lower average ages compared to their non-Web-based control groups.

Dose of an Intervention

There are tools available that ascertain use of a Web site, visits to a various pages on the site, and paths to trace links and usage patterns by the user. These are useful to determine the dose of the Web-based intervention. Based on the individual's response, how much intervention that is needed by an individual can be tailored and varied. In the reviewed studies that discussed their Web site use statistics, (see Table 4) there was large variability in the average intervention time and the number of logons to the sites. The average session site time of 19.3 minutes should be considered in context of the attributes of the individual using the Web site and the burden the intervention may place on the individual to complete the items and contribute any necessary interactive responses. The burden to complete the needed information throughout the site may be relieved by increased interactivity to create and maintain interest in the site. Interactivity may help reduce attrition of Web users and provide benefits in producing positive behavioral change.