Participants were recruited through advertisements in newspapers and via the Internet. To be included in the study, participants had to be aged 18 years or older, presenting with depressive symptoms, and willing to participate in a self-help course. Presence of depressive symptoms was ascertained with the Center of Epidemiologic Studies Depression scale (CES-D) when people scored 16 or higher on this scale [15]. No other inclusion or exclusion criteria were defined for this study.

A total of 263 participants were randomized to one of the three conditions: Internet-based cognitive behavioral therapy (CBT, n = 88), Internet-based problem-solving therapy (PST, n = 88) and a waiting list control group with unrestricted access to usual care (WL, n = 87). Block randomization was used, with each block containing 9 allocations. An independent statistician conducted the randomization. The study protocol was approved by the Medical Ethics Committee of the VU University Medical Centre.

The problem-solving therapy-intervention was a Dutch adaptation of Self-Examination Therapy by Bowman [16]. Problem-solving therapy consisted of 3 steps. First, the participants made a list of the most important things in their lives. Second, they wrote down their current worries and problems and divided these into important and solvable problems, important but unsolvable problems, and unimportant problems. The core element of problem-solving therapy is to address the solvable important problems following a six-step procedure: describing the problem, brainstorming, choosing the best solution, making a plan for carrying out the solution, actually carrying out the solution, and evaluation. During the third step, the participants made a plan for the future in which they described how they would try to accomplish those things that matter most to them. The course took five weeks and consisted of one lesson a week.

The cognitive behavioral therapy intervention was based on the “Coping with Depression” course [17], Dutch version [18]. Cognitive behavioral therapy in this study included psycho-education and focused on skills such as relaxation, cognitive restructuring (including coping with worrying thoughts), social skills training, and behavioral activation, specifically increasing the number of pleasant activities. Cognitive behavioral therapy consisted of 8 lessons, 1 lesson a week followed by a booster session after 12 weeks.

Participants in both interventions were supported by a life coach via email during the intervention period. Support was directed at helping the participant to work through the intervention and not at developing a therapeutic relationship or giving advice on how to cope with depressive symptoms or other problems. The average time that therapists spent on each participant providing feedback and answering questions via email was estimated at 20 minutes per week.

All analyses were performed in accordance with the intention-to-treat principle. The expectation maximization algorithm (EM) in Statistical Package for the Social Sciences (SPSS, version 17) (SPSS, Inc, Chicago, IL, USA) was used to estimate missing values. EM employs maximum likelihood estimation to replace missing values in the data set with estimates.

The average age of the 263 participants at baseline was 45 years (SD 12.1). Most participants were female (187/263, 71%) and the majority had higher vocational education/university (168/263, 64%) or intermediate vocational education/high school (72/263, 27%). Almost all participants were Dutch (243/263, 92%). The mean score of the participants on the CES-D was 31.7 (SD 7.5, median 31.0) at baseline. Mean utility score on the EQ-5D was 0.61 (SD 0.22). There were no statistically significant differences between the three groups with respect to the demographics, depressive symptoms, or quality of life scores at baseline [13]. At baseline, total per capita costs were € 846, € 893, and € 925 for the cognitive behavioral therapy, problem-solving therapy, and waiting list groups respectively. No significant differences were found in total costs between the three groups, F _{2, 262} = 0.10, P = .91.

After 12 weeks, cognitive behavioral therapy and problem-solving therapy resulted in significantly higher quality of life scores and lower depression scores than waiting list placement. Full details of the main clinical outcomes are reported elsewhere [13].

The mean number of QALYs during the period of 12 weeks was 0.16 for cognitive behavioral therapy (95% CI 0.152 – 0.169), 0.16 for problem-solving therapy (95% CI 0.152-0.168), and 0.15 for wait-list (95% CI 0.142-0.159). No significant differences were found between groups, F _{2, 262} = 1.83, P = .16. Regarding clinically significant change, both cognitive behavioral therapy and problem-solving therapy tended to show more clinically significant change than waiting list placement after 12 weeks, χ² _{2, 263} = 5.10, P = .08. The number of participants showing clinically significant change was 25 (28.4%) for cognitive behavioral therapy, 23 (26.1%) for problem-solving therapy, and 13 (14.9%) for waiting list placement.

Cost data were available at baseline from 252 participants (cognitive behavioral therapy, n = 83; problem-solving therapy, n = 85; waiting list group, n = 84). At 12-weeks follow-up, 147 participants returned cost data (cognitive behavioral therapy, n = 45; problem-solving therapy, n = 40; wait-listed group; n = 62). For descriptive purposes, Table 2 presents the number of participants using health care services with productivity costs due to production losses, based on the sample of participants that returned cost data at baseline and at 12-weeks follow-up.

At baseline, more than half of the participants used some form of medication. This level of medication use continued into the follow-up period. Of all participants, 30% to 40% had contact with their general practitioner at both assessments, which happened to the same degree across all conditions. Reasons for visiting the general practitioner were, however, unknown. Frequently used services other than the general practitioner included visits to psychotherapists with a private practice, medical specialists, physiotherapists, and regional mental health services. At follow-up, visiting the private practice psychotherapist was reduced only in the problem-solving therapy group. The use of regional mental health services was reduced in both intervention groups.

With regard to production losses, a majority of the participants reported domestic costs at baseline with evident reductions reported by both intervention groups at follow-up. Reductions were also reported by both intervention groups in the number of participants that incurred costs due to work loss and work cutback in paid employment.

Table 3 presents costs for the different cost categories based on imputed data. The majority of total costs were attributable to the indirect (productivity) costs. Indirect costs were, on average, lower by € 201 (€ 1900-€ 1701) and € 258 (€ 1900-€ 1642) for cognitive behavioral therapy and problem-solving therapy, respectively, relative to waiting list placement. The differences in total direct medical costs were largely due to the inclusion of intervention costs for both cognitive behavioral therapy and problem-solving therapy. Overall, the cognitive behavioral therapy group incurred extra costs of € 256 (€ 2814-€ 2558) compared with the waiting list group. Problem-solving therapy led to extra costs of € 147 (€ 2705-€ 2558) compared with waiting list placement. Total costs between groups were not significant, F _{2, 262} = 0.15, P = .86.

Cost-utility analysis showed that the median incremental cost-effectiveness ratio for cognitive behavioral therapy versus waiting list placement resulted in € 22,609 per QALY gained. Hence, for each QALY gained by offering cognitive behavioral therapy instead of waiting, extra costs of € 22,609 are incurred. The left-hand panel of Figure 1 presents the distributions over the cost-effectiveness plane for the comparisons between the 2 active interventions and waiting list. Of the simulated ICERs, 28% are in the southeast quadrant, indicating a 28% probability that cognitive behavioral therapy is a better treatment because it generates more health effects for lower costs when compared with a waiting list control group. However, the majority of the simulated ICERs (67%) occur in the northeast quadrant, indicating that a health gain is produced, but at additional costs. Table 4 presents the median ICERs and their confidence intervals.

The median ICER for problem-solving therapy versus waiting list placement resulted in extra costs of € 11,523 per QALY gained by offering problem-solving therapy instead of waiting. In addition, there is a 58% probability that it is more effective in terms of QALYs at extra costs and there is a 37% probability that it is more effective at a lower cost.

Problem-solving therapy has a 35% probability of being the dominant treatment. And half of the ICERs are equally distributed over the northwest quadrant (24%) and the southwest quadrant (27%). The fact that the ICERs are almost equally divided over the four quadrants indicates no obvious preference for 1 of the 2 active interventions (Figure 3, left-hand).

The right-hand panel of Figure 1 presents the acceptability curves for the comparisons between the 2 active interventions and waiting list placement. Regarding the active interventions versus waiting list placement, there is a 28% and 38% probability respectively that cognitive behavioral therapy and problem-solving therapy are more cost-effective than waiting, if society places a zero value on one gained QALY. However, the probability of Internet-based therapy being more cost-effective increases when the willingness to pay for a QALY gained increases. To illustrate, with a willingness to pay ceiling of € 30,000 per gained QALY, then cognitive behavioral therapy and problem-solving therapy have a probability of 52% and 61% respectively of being more cost-effective compared with waiting.

In the cost-effectiveness analysis, the incremental cost-effectiveness ratio is expressed in terms of additional costs per clinically significant change in depressive symptom severity. The median ICER for cognitive behavioral therapy versus waiting list placement resulted in € 1817, indicating that by offering cognitive behavioral therapy instead of placing participants on a waiting list, extra costs of € 1817 are incurred for a health gain of one additional reliably improved participant. There is a 69% probability that a participant will change with cognitive behavioral therapy, but at additional costs. The distributions of the bootstrapped ICERs over the cost-effectiveness plane for the comparisons between the 2 active interventions and waiting list placement are presented in the left-hand panel of Figure 2. For problem-solving therapy versus waiting list placement, ICER is € 1248 per reliably improved participant. The probability that problem-solving therapy is more effective in terms of reduced depression severity but more costly than waiting is 60%. Comparing problem-solving therapy with cognitive behavioral therapy resulted in a median ICER of -36. As for cost-utility, the four quadrants each contain an almost equal percentage of the ICERs, meaning no evident preference for 1 of the 2 Internet-based treatments (Figure 3, right-hand).

The acceptability curve for cognitive behavioral therapy versus waiting list placement showed that with no willingness to pay for one reliably improved participant, there is a 30% probability that cognitive behavioral therapy is more cost-effective than waiting (Figure 2, right-hand panel). However, the probability rapidly increases when people are willing to pay more for clinically significant improvement. With a willingness to pay of € 5000 and € 10 000, cognitive behavioral therapy has a probability of 75% and 91% respectively of being more cost-effective compared with waiting list placement. For problem-solving therapy versus waiting list placement, the probability that problem-solving therapy is more cost-effective is 38% in the case of no willingness to pay. However, WTP of € 5000 and of € 10 000 result in probabilities of 75% and 89%.

Cost-utility and cost-effectiveness analyses were performed using multiple regression as an imputation method for missing observations. Regarding cost-utility, regression imputation led to better results for both interventions compared with the main analysis (where maximum likelihood estimation was used), the probabilities that the interventions result in better outcomes at lower costs than waiting list placement are 48% for cognitive behavioral therapy (was 28% when using maximum likelihood estimation) and 79% for problem-solving therapy (was 37% when using maximum likelihood estimation). For the comparison between the two active treatments, results are in favor of problem-solving therapy with a 72% probability that problem-solving therapy leads to better outcomes in terms of QALYs at lower costs. Cognitive behavioral therapy and problem-solving therapy have high probabilities (73% and 95% respectively) of being acceptable treatments with a WTP ceiling of € 30 000 (Figure 1, right-hand panel).

In the cost-effectiveness analyses, regression imputation also resulted in higher probabilities for the interventions being cost-effective compared with waiting list placement. In this case, the probabilities that the interventions will result in more clinically significant change at lower costs are 50% for cognitive behavioral therapy (was 30% when using maximum likelihood estimation) and 80% for problem-solving therapy (was 36% when using maximum likelihood estimation). And problem-solving therapy has a high probability (77%) of producing worse clinical outcomes at lower costs than cognitive behavioral therapy. With a WTP ceiling of € 5000, the probability of cognitive behavioral therapy being acceptable is 99% from a cost-effectiveness perspective (Figure 2, right-hand panel), while for problem-solving therapy this probability is 80% in the case of no willingness to pay.

At first glance, cost-utility analyses produced conservative results regarding the efficiency of Internet-based treatment. With no willingness to pay for one extra quality-adjusted life-year, cognitive behavioral therapy and problem-solving therapy had a probability of only 28% and 38% of being more acceptable than waiting. But generally, people are willing to pay for one extra quality-adjusted life-year, although there is no consensus regarding the value of one QALY [25]. When the willingness to pay was raised to € 30 000, then the Internet-based interventions had a 52% and 61% probability of being the preferred option. With the same threshold of € 30,000, sensitivity analyses based on another way of imputing missing observations showed similar or higher probabilities (≥ 73%) of cognitive behavioral therapy and problem-solving therapy being acceptable treatments, thus attesting to the robustness of our findings. From a cost-effectiveness perspective, both treatments showed high probabilities of being more cost-effective compared with waiting even when society might place modest values on one clinically significant improved patient; cognitive behavioral therapy and problem-solving therapy showed a 91% and 89% probability respectively of being more cost-effective than waiting when society was assumed to have been prepared to pay € 10,000 for recovery from depression. Comparing both Internet-based treatments indicated no obvious preference for one of the treatments.

Only a small number of depressed people are reached within traditional health care settings [26]. As Internet-based treatments are more accessible than traditional therapies, these treatments are assumed to be able to reach more people in need of treatment. Internet-based treatments for depression, mostly based on cognitive behavioral therapy, have been shown to be effective [12]. An important further step is to assess the cost-utility and cost-effectiveness of Internet-based treatments. Cost-utility analysis allows health care organizations and other stakeholders to compare the benefits of online treatment with other interventions across a range of disorders with which treatments for depression may be in competition for limited resources. The study of McCrone et al (2004) made a start by showing that computer-delivered cognitive behavioral therapy had a high likelihood of being cost-effective in primary care patients. Our study showed that two different Internet-based treatments with modest levels of therapist support were cost-effective in reducing depressive symptom severity in the general population, even in the conservative scenario that society had a limited willingness to pay for a reliably improved patient.

From a cost-utility point of view, the acceptability of the two treatments depends to a larger extent on what society is prepared to invest to achieve one quality-adjusted life-year. Differences in results between cost-utility and cost-effectiveness results can be explained by the nature of the outcomes. The quality-adjusted life-year is a measure of disease burden, including both the quality and the quantity of life lived, covering diverse health state dimensions, including mental health. The interventions were, however, directed at improving mental health, that is, reducing depression, and a depression-specific measure was used. Therefore, it cannot be expected that improvement in depression will result in across-the-board improvement in quality of life.

Besides, this study showed that a brief intervention based on problem-solving therapy seems to be a good alternative for Internet-based cognitive behavioral therapy in terms of cost-effectiveness. The generic nature of problem-solving therapy makes it suitable as a first step in a stepped care model. This would enable therapists to free up their limited resources and direct these to people presenting with more complex and severe symptomatology.

We acknowledge the following limitations of this study. First, we were faced with a high attrition rate. The use of maximum likelihood estimation to handle missing data might have introduced some bias. It is, however, a highly recommend method [27]. In a sensitivity analysis, outcomes and costs were, in addition, calculated using multiple regression as an imputation method for missing observations. Cost-effectiveness evaluations then became more favorable for both interventions. This implies that with high attrition rates, results partly depend on the imputation method used. For our data, it was understood that maximum likelihood estimation was the more conservative method; it was also the method used in the main analysis.

Second, the costs and effects were considered in the relatively short time span of 12 weeks. Accordingly, we do not know how the cost-effectiveness of Internet-based treatment is affected when a longer period is used, and this is a question that remains open for further research. Third, the study was not powered to detect statistically significant differences in outcomes and costs between the two Internet-based interventions. Therefore results regarding the comparison between the two active interventions should be considered explorative. Fourth, we used self-report for all measures. Self-report can be vulnerable to recall bias. The self-report of medication and care products for example, is often underestimated [28].

In light of these limitations, our findings should be interpreted with some caution.