Critical, evidence-based evaluation of the effectiveness of information and communication technology should be one important component of eHealth [
In this issue of the Journal, researchers at the Kaiser Permanente Center for Health Research present results of a study to develop and evaluate a web-based psychoeducational program designed to reduce depression [
Two hundred and ninety-nine patients were randomly selected to receive either the online depression program in addition to usual care, or usual care alone. Previous studies evaluating traditional bibliotherapy (the treatment of depression with educational material) suggest that bibliotherapy is an effective intervention. Despite the fact that the Internet offers various possibilities to improve traditional educational and bibliotherapy material, the trial disappointingly did not result in any measurable effect in the overall sample. Although in an exploratory data analysis researchers found a small benefit for those participants who entered the study with lower levels of depression, these results need to be confirmed by subsequent studies. "We are at the very threshold of this burgeoning field, and we know very little about the circumstances and processes that will optimize the delivery and acceptance of these interventions", write Greg Clarke and colleagues. "There is no accumulated clinical lore about how to best provide Internet services; we are blazing this trail as we progress."
The study of Clarke and colleagues illustrates the possibilities and challenges of evaluating the impact of a health website on health outcomes in a web-based RCT.
Among the appealing factors is the fact that hundreds or thousands of people per day may frequent a health website, which gives plenty of sample size. Patients from all over the world can be recruited, enhancing the external validity (generalizability) of the results. No costly face-to-face interactions such as clinical examinations may be necessary if psychological outcomes are measured through self-administered electronic questionnaires. The administration of the intervention or the control, respectively, data collection and outcome measurement can be completely automatized. Not only does this make web-based RCTs very cheap, it minimizes biases, at least those introduced by human observers. Often an investigator's earliest opportunity to interact with the research subjects is when he opens the database to analyze the data.
On the other hand, there are significant challenges inherent in web-based randomized trials that don't exist in clinical trials studying drugs. Some of them are similar to challenges in educational interventions or surgical trials: For example, the trial cannot be conducted in a double-blind fashion as the patient always knows what intervention he receives.
But there are perhaps even more serious challenges. First, there is a considerable risk that the control group becomes "contaminated" by accessing a similar intervention from somewhere else on the web. This is particularly true if the intervention is "giving information" or an educational program that - in a similar way - can be easily found somewhere else on the web. Interventions such as smoking cessation programs can be easily found and used elsewhere on the web, threatening the ability to detect differences between the groups. Institutional review boards may require investigators to describe their intervention in detail before patients consent to participate. Participants who are randomized into the control group may be disappointed that they are not getting the intervention and may search the web for a comparable intervention, using it without the knowledge of the investigator. To minimize this bias the only option is to reduce the amount of information about the intervention given to participants, which may be ethically problematic. One should also ask participants in the control group whether they used similar interventions elsewhere on the web, or even monitor their use of other websites directly by using client-side proxy software.
Apart from the problem of ensuring that the control group actually stays a control group, we are facing the opposite problem in the intervention group: How do we assure that the intervention group is actually using the intervention? The relative ease of enrolling participants to a web-based trial seems to come at the cost of a high probability to lose them again - as many as about half of the patients [
The bias introduced by these issues is typically a "bias towards the null", i.e. through these methodological difficulties a trial may fail to show a small effect of an intervention, through the "noise" introduced. Despite these issues and despite alternative possibilities to evaluate a website (surveys, log-file analysis, before-after trials, and interrupted time series) the RCT remains the gold standard and we are eagerly looking forward to see more of these trials.