This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Depression is common and treatable with cognitive behavior therapy (CBT), for example. However, access to this therapy is limited. Internet-based interventions have been found to be effective in reducing symptoms of depression. The International Society for Research on Internet Interventions has highlighted the importance of translating effective Internet programs into multiple languages to enable worldwide dissemination.
The aim of the current study was to determine if it would be cost effective to translate an existing English-language Internet-based intervention for use in a non-English-speaking country.
This paper reports an evaluation of a trial in which a research group in Norway translated two English-language Internet-based interventions into Norwegian (MoodGYM and BluePages) that had previously been shown to reduce symptoms of depression. The translation process and estimates of the cost-effectiveness of such a translation process is described. Estimated health effect was found by using quality-adjusted life years (QALY).
Conservative estimates indicate that for every 1000 persons treated, 16 QALYs are gained. The investment is returned 9 times and the cost-effectiveness ratio (CER) is 3432. The costs of the translation project totaled to approximately 27% of the estimated original English-language version development costs.
The economic analysis shows that the cost-effectiveness of the translation project was substantial. Hopefully, these results will encourage others to do similar analyses and report cost-effectiveness data in their research reports.
Computer-aided psychotherapy is a promising way to increase accessibility to evidence-based treatment of many mental disorders, such as mood disorders [
In a systematic review of computer-aided psychotherapy programs, Marks et al [
A recent quantitative meta-analysis on Internet and computerized interventions for adult depression found an overall effect size of
In their meta-analysis, Andersson and Cuijpers [
Internet-based self-help has the potential to reach target groups with an unmet need for help. In a recent study, we found that two-thirds of the participants completing a trial of an Internet-unguided cognitive behavioral therapy (CBT) program initially reported an unmet need for help with a psychological problem [
There is now sufficient evidence to suggest that Internet interventions can be effective in the prevention and treatment of depression [
The aim of this study is to evaluate the translation process of an Internet-based self-help intervention up to dissemination and to study cost and cost-effectiveness of the project. It will also consider the feasibility of providing access to Internet-based interventions in the national language versions. Further, we discuss some of the challenges experienced during this project and give recommendations for future developments. Finally, we attempt to demonstrate the use of disease-specific scales to estimate quality of life.
Before translating the Internet-based interventions, a study was undertaken to investigate the need for a Web-based self-help intervention [
In February 2006, Norwegian language versions of BluePages and MoodGYM were made available. The planning of this project started in July 2004 and the project was completed in June 2006. The translation of BluePages was formalized in a licensing deed, specifying ANU as the intellectual property owner and the obligations for the collaborating partners. For MoodGYM, the research collaboration agreement specified a nonexclusive, nontransferable license for the translated version during the period of collaboration and joint ownership of the data emerging from the collaboration. The translation of MoodGYM and BluePages was carried out between October 2005 and January 2006. The Norwegian version of the BluePages website was developed by using a 2-phase process. First, a professional translator prepared a Norwegian version of BluePages. In the second phase, the translation was adjusted by the research group at the University of Tromsø in Norway to ensure that it was culturally and clinically appropriate for Norwegian users. The MoodGYM training program was translated in 4 phases. The research group conducted the first phase of translation. In phase 2, clinical professionals with formal competence in cognitive therapy scrutinized the translation from the first phase and made adjustments to the text when necessary. In phase 3, an expert translator of English compared the Norwegian version of the program with the original Australian version and checked for inconsistencies. In the fourth phase, the research group evaluated all changes and finalized the translation.
The Norwegian versions of MoodGYM and BluePages were evaluated in a RCT that compared the effect on depressive symptoms of an unguided Internet-based intervention. The Internet condition consisted of a depression information website and a self-help Web application that delivered automated CBT. The participants in the waiting list condition were free to access formal or informal help as usual. This trial was organized as an unguided quasi-indicated prevention intervention. A total of 163 students (mean age 28.2 years) with elevated psychological distress were enrolled into the trial. The intent-to-treat effect size for depressive symptoms was
The total cost estimate for the Norwegian version of MoodGYM and BluePages was estimated from the cost of the translation process and the project costs for the Norwegian evaluation trial.
The translation process was accomplished by using 2 members from the Norwegian research group, 3 students in their final year of clinical psychology training, and external resources (translators, psychiatrists, mailing/printing services). Translation costs for employees and students were calculated based on the estimated time consumed and their salaries. The cost for external resources was based on payments for their services.
The trial costs were incurred over 8 weeks from initial participant contact until the completion of the validation trial. This estimate for total costs was based on all expenditures of the validation trial for BluePages and MoodGYM and included the direct costs of mailing and printing during the trial, and also salary for employees and students. Working hours for developing funding applications were not included in the estimates. Because the aim of this study was to evaluate the translation process up to dissemination, the trial costs are included even though these costs could be regarded as research costs.
The initial cost data were in Norwegian kroner (NOK) and dated back to late 2005 and early 2006. For the purpose of this paper, the cost has been adjusted for inflation based on the Norwegian consumer price index into a 2009 price level [
The Center for Epidemiologic Studies Depression Scale (CES-D) [
The Kessler Psychological Distress Scale (K10) [
There has been a growing interest for applying measures for health-related quality of life (QOL) to evaluate health care services, in general. However, in the field of Internet- and computer-based self-help for mental disorders, reviews find that studies lack cost-effectiveness data [
The Rosser classification of illness states scale (Rosser Index) [
In the present study, the Rosser Index was used as a generic instrument to evaluate the observed effects for the Internet intervention. The method is based on classifying outcome data from the trial into 2 components: disability categories (I-VIII) and distress categories (A-D) that define 29 potential health states [
The original valuation matrix was based on 70 respondents (doctors, nurses, patients, etc) and was not a random sample of the population. The matrix has been transformed and validated in several publications [
Model for the Rosser Index measuring quality of life (QOL). The health state matrix is based on the Center for Epidemiologic Studies Depression scale (CES-D) scores in relation to the Rosser Disability Category and the Kessler Psychological Distress Scale (K10) scores in relation to the Rosser Distress Category.
The CES-D was used as the basis for deciding the appropriate disability categories. No other studies were found that used CES-D as the disability categories in the Rosser health state matrix. The authors have functioned as a reference group of clinical psychologists and found that the CES-D seems to correspond to the Rosser disability dimensions (see
Center for Epidemiologic Studies Depression scale (CES-D) scores and their corresponding Rosser disability categories.
CES-D | Rosser disability category | ||
Subgroup | Score | Category | Description |
Subclinical | 0-15 | I | No social disability |
Mild/moderate | 16-23 | II | Slight social disability, can continue almost as usual with occupational and home activities |
Moderate/severe | 24-33 | III | Severe social disability and/or slight impairment of performance at work |
|
34-42 | IV | Choice of work or performance at work very severely limited |
|
43-52 | V | Unable to undertake any work/education |
|
53-60 | VI | Full-time care or in an institution |
The K10 was developed for use in epidemiological surveys and is used to measure psychological distress. No other studies were found that used K10 as the distress category in the Rosser health state matrix. The authors have functioned as a reference group of clinical psychologists and found that the K10 seems to correspond largely with the Rosser distress dimensions (see
Kessler Psychological Distress Scale (K10) scores and their corresponding Rosser distress categories.
K10 | Rosser distress category | ||
Subgroup | Score | Category | Description |
Being well | 10-19 | A | No distress |
Mild mental disorder | 20-24 | B | Mild distress |
Moderate mental disorder | 25-29 | C | Moderate distress |
Severe mental disorder | 30-50 | D | Severe distress |
A QALY is a year of life adjusted for its quality or its value. A year in perfect health is considered equal to 1 QALY [
The National Institute for Health and Clinical Excellence [
The Norwegian real cost totalled to €39,900, and the estimated cost to €56,000, adding up to a total of €95,900. Based on data published by Butler and colleagues [
Completers between-group effect size (Hedges’
CES-D scale and condition | Pretest | Posttest | Contrasta | Effect sizeb | |
|
Mean (SD), n | Mean (SD), n | Mean (SD) |
|
|
|
|
|
|
|
|
|
Intervention | 22.6 (10.9), 43 | 18.5 (14.0), 43 | 4.1 (10.4) | 0.72 (0.21) |
|
Control | 18.5 (9.6), 59 | 21.4 (13.0), 59 | –3.0 (9.1) |
|
|
|
|
|
|
|
|
Intervention | 11.4 (3.6), 14 | 10.0 (8.6), 22 | 1.4 (8.4) | 0.68 (0.34) |
|
Control | 9.8 (3.4), 25 | 14.4 (9.7), 25 | –4.6 (8.8) |
|
|
|
|
|
|
|
|
Intervention | 19.0 (2.7), 10 | 11.0 (9.5), 7 | 8.0 (9.8) | 0.85 (0.42) |
|
Control | 19.2 (2.3), 17 | 19.5 (10.9), 11 | –0.3 (9.2) |
|
|
|
|
|
|
|
|
Intervention | 32.7 (7.0), 19 | 28.6 (12.8), 14 | 4.1 (11.7) | 0.67 (0.34) |
|
Control | 30.4 (10.0), 17 | 32.4 (12.1), 23 | –3.2 (9.2) |
|
a Positive contrast represents better outcome.
b Hedges’
The following analysis is based on the completers data from the effect trial [
Descriptive matrix with number of participants for each disability and distress state, for condition and test time.
Condition | Disability category | Distress category, n | |||||||
|
|
Baseline | Posttest | ||||||
|
|
A | B | C | D | A | B | C | D |
Internet | I | 1 | 8 | 5 |
|
1 | 12 | 8 | 1 |
|
II |
|
6 | 3 | 1 |
|
4 | 2 | 1 |
|
III |
|
3 | 4 | 3 |
|
2 | 1 | 2 |
|
IV |
|
2 |
|
5 |
|
1 | 1 | 5 |
|
V |
|
|
|
2 |
|
|
|
2 |
Control | I | 1 | 11 | 10 | 3 |
|
10 | 12 | 3 |
|
II |
|
7 | 4 | 6 |
|
6 | 3 | 2 |
|
III |
|
1 | 7 | 6 | 1 | 3 | 1 | 5 |
|
IV |
|
1 |
|
|
|
1 | 5 | 3 |
|
V |
|
1 |
|
1 |
|
|
|
3 |
|
VI |
|
|
|
|
|
1 |
|
|
Rosser´s utility valuation scores for health conditions.
Disability category | Distress categorya | |||
|
A | B | C | D |
I (no social disability) | 1.000 | 0.995 | 0.990 | 0.967 |
II (slight social disability) | 0.990 | 0.986 | 0.973 | 0.932 |
III (severe social disability) | 0.980 | 0.972 | 0.956 | 0.912 |
IV (severely limited work performance) | 0.964 | 0.956 | 0.942 | 0.870 |
V (unable to work/study) | 0.946 | 0.935 | 0.900 | 0.700 |
VI (total social disability) | 0.875 | 0.845 | 0.680 | 0.000 |
a 1 = healthy; 0 =dead. Table shows only relevant disability categories (for complete table see Kind et al [
When all cells in the descriptive matrix are multiplied with the corresponding weight in the valuation matrix, the sum of all these products gives the total QOL score for condition and time. The total QOL score is then divided by the number of participants in the group, which gives an average QOL gain (∆H) score for the group [
By using the change scores from the Australian 12-month follow-up study of MoodGYM and BluePages [
The gain in QOL (∆H) for the Internet intervention group was 0.0048, and increased to 0.0054 (0.0048×1.127) after transformation. The negative gain in ∆H for the control group was –0.015 and decreased to –0.013 after transformation. The between-groups ∆H reduced to 0.018. The transformation extrapolates the results to a timeframe of 1 year. This gives a QALY-gain=0.018×1, which equals 0.018 of a QALY.
Estimated development and translation costs with mean cost (based on 3 years’ operating time).
Project and costs type | Costs per year (€1000) | Mean cost (€1000) | |||
|
|
Year 1 | Year 2 | Year 3 |
|
|
|
|
|
|
|
|
Development | 479.4 |
|
|
159.8 |
|
Maintenance | 44.7 | 44.7 | 44.7 | 44.7 |
|
Total | 524.1 | 44.7 | 44.7 | 204.5 |
|
|
|
|
|
|
|
Development | 95.9 |
|
|
32.0 |
|
Maintenance | 14.0 | 14.0 | 14.0 | 14.0 |
|
Service fee | 9.0 | 9.0 | 9.0 | 9.0 |
|
Total | 118.9 | 23.0 | 23.0 | 55.0 |
Based on data from the validation study, the QALY-gain was found to be 0.018 of a QALY. To obtain one QALY, we need 56 individuals (1/0.018) to use the Internet intervention in the same way as the group of completers did in the trial. This is equal to gaining 1 year of full health for 1 person.
Estimates for cost and annual savings (based on 3 years’ operating time) for development and translation projects, with cost-effectiveness ratio (CER).
Project estimate | Per QALY | Annual | |||||||
|
|
Savings |
Persons | Development cost (€1000) | Persons treated | Savings per persona
|
Total savings |
QALYsb | CERc |
|
|
|
|
|
|
|
|
|
|
|
Norwegian | 67 | 56 | 204 | 1000 | 0.99 | 992 | 15 | 13,772 |
|
Norwegian | 67 | 56 | 204 | 20,000 | 1.19 | 23,725 | 354 | 576 |
|
UK | 30 | 56 | 204 | 1000 | 0.33 | 332 | 11 | 18,450 |
|
UK | 30 | 56 | 204 | 20,000 | 0.53 | 10,510 | 350 | 582 |
|
|
|
|
|
|
|
|
|
|
|
Norwegian | 67 | 56 | 55 | 1000 | 1.20 | 1141 | 17 | 3228 |
|
Norwegian | 67 | 56 | 55 | 20,000 | 1.20 | 23.875 | 356 | 154 |
|
UK | 30 | 56 | 55 | 1000 | 0.54 | 481 | 16 | 3432 |
|
UK | 30 | 56 | 55 | 20,000 | 0.54 | 10,659 | 355 | 155 |
a Savings per person = (savings per QALY)/(persons per QALY)–(development cost)/(persons treated).
b QALYs = (total savings)/(savings per QALY).
c CER = (development cost)/QALYs.
Our model for estimating the gain in QOL is based on the Rosser Index. Depending on how well the CES-D and K10 scales were fitted to the disability and distress categories, this could affect the output from the model. A sensitivity analysis was conducted to explore how the operationalizing of the scales could have contributed to variation in the outcome (QOL gain). Three scenarios were investigated: (1) the main analysis based on the score for each individual participant, (2) by using the CES-D subgroup mean, and (3) by using the mean for the conditions. In addition, 2 scenarios were investigated to explore the effect of the deterioration in the control group: (4) if there was no change in the control group, and (5) a positive change in the control group. Scenario 1 is already described previously.
Scenario 2 had only 3 values for each condition and time. In the Internet condition, scores for CES-D and K10 placed the CES-D subgroups as follows at pretest and posttest: subclinical group (disability I/distress B, K10 mean 22.8; disability I/distress B, K10 mean 23.7), the mild/moderate subgroup (disability II/distress B, K10 mean 24.4; disability II/distress C, K10 mean 25.3), and the moderate/severe subgroup (disability III/distress D, K10 mean 30.0; disability III/distress D, K10 mean 30.7). In the control condition, scores for CES-D and K10 placed the CES-D subgroups as follows at pretest and posttest: subclinical group (disability I/distress B, K10 mean 24.5; disability I/distress C, K10 mean 25.4), the mild/moderate subgroup (disability II/distress C, K10 mean 27.5; disability II/distress C, K10 mean 25.1), and the moderate/severe subgroup (disability III/distress C, K10 mean 28.5; disability III/distress C, K10 mean 28.3).
Scenario 3 was placed in the disability category II for both conditions, based on CES-D scores (
Scenario 4 aimed to explore the effect of the deterioration in the control group. If there were no change in the control group during the trial, the QOL gain for the intervention would be equal to the gain for the Internet intervention group. The QOL gain between groups is 0.006, only one-third of the QOL gain from scenario 1.
Scenario 5 represents a scenario in which the control group had a positive change from baseline to posttest, as one could expect during a time period of 12 months. This scenario uses the pretest assessment for both conditions to calculate a new factor to extrapolate our findings to predict a 12-month outcome. The extrapolating was done directly on the pretest scores for each participant to predict outcome scores at 12 months. The rationale for exploring this is based on the fact that the Norwegian and Australian trials were different, unattended vs minimal contact/attention placebo. The change factors for the Internet intervention and control group were 1.546 and 1.317, respectively. The QOL gain between groups is 0.012, ie, smaller than for scenario 1 (two-thirds), but still notable. An alternative approach for this scenario could be to extrapolate the Norwegian posttest scores and compare to the Australian improvement for the control group. The change factors for the Internet intervention and control group would then be 1.127 (as for scenarios 1-3) with a QOL gain of 0.003. This is only one-sixth of the QOL gain from scenario 1. See
Quality of life (QOL) sensitivity analysis within and between conditions.
Scenario | Condition | ∆Ha | ∆Hext b |
1. Main analysis |
|
|
|
|
Internet QOL gain | 0.005 | 0.006 |
|
Control QOL gain | –0.015 | –0.013 |
|
Between-conditions QOL gain | 0.020 | 0.018 |
2. Subgroup mean |
|
|
|
|
Internet QOL gain | 0.008 | 0.010 |
|
Control QOL gain | –0.004 | –0.003 |
|
Between-conditions QOL gain | 0.012 | 0.013 |
3. Condition mean |
|
|
|
|
Internet QOL gain | 0.020 | 0.024 |
|
Control QOL gain | –0.020 | –0.017 |
|
Between-conditions QOL gain | 0.040 | 0.041 |
4. Main analysis, unchanged control group | |||
|
Internet QOL gain | 0.005 | 0.006 |
|
Control QOL gain | 0.000 | 0.000 |
|
Between-conditions QOL gain | 0.005 | 0.006 |
5. Main analysis, gain in control group | |||
|
Internet QOL gain |
|
0.024 |
|
Control QOL gain |
|
0.012 |
|
Between-conditions QOL gain |
|
0.012 |
a ∆H=QOL gain.
b ∆Hext = QOL gain extrapolated.
Conservative estimates indicate that for every 1000 persons treated, 16 QALYs are gained. The investment is returned 9 times and the CER is 3432. The costs of the translation project totalled to approximately 27% of the estimated original English-language version development costs.
The scenarios in the sensitivity analyses differ in that they decrease in distribution among the 29 potential health states. Scenario 1 uses up to 15 different states, whereas scenario 2 uses up to 5 different states, and scenario 3 only 1 health state. As the distribution becomes more even, the QOL gain (∆H) between groups decreases, and for scenario 3 there is no gain at all. This is partly an effect of the negative results for the control condition. If the control group is kept stable (no change between pretest and posttest score), the QOL change between groups is equal to the change within the intervention condition. Then there is an opposite effect, in which the QOL gain increases with the decrease in distribution. The result is less sensitivity and gives an inflated result. Between scenarios 1 and 2, this is probably a result of less variance and not necessarily that we had to use judgment, as in scenario 3. The results for the extrapolated QOL gain (∆Hext) offer no additional insight because it is just a fixed factor that scales the scores into a 12-month QOL gain.
Scenarios 4 and 5 explore the effect of the deterioration in the control group, as one could expect some improvement in that group over a longer time period. The Australian trial is not comparable to the Norwegian because the experimental and control groups are given different focus (minimal contact vs unattended). As scenario 4 shows, we need 3 times as many persons treated to gain 1 QALY. In scenario 5, we need 1.5 as many persons treated to gain 1 QALY.
As the economic analysis shows, the cost of the translation project is less than 1 gained QALY. In the worst case, it is necessary to treat 46 subjects with symptoms of depression to reach the cost break-even point. The treatment received should, on average, be the same as the average for the completers in the trial. As a translation project, this investment is highly cost-effective as it returns the investment many times. With the Norwegian estimate for gained QALY (savings per QALY), every 1000 persons treated yields 17 QALYs and returns the investment 21 times with a CER at 3228. Based on the UK estimate for gained QALY, every 1000 persons treated gives 16 QALYs, returns the investment 9 times, and the CER is 3432. The cost for the translation projects is less than 1 QALY, so the annual number needed to treat to make the cost break-even is 46 individuals, and the development project needs 171 individuals (based on Norwegian estimate for savings per QALY). This is promising, as we know that people, in general, are positive to Internet-based self-help and that this intervention can serve as a prevention intervention. Results from a meta-analysis show that preventive interventions can reduce the incidence of depressive disorders by 22% [
The trial costs are included in development costs, and include testing the Web-based interventions’ effect and effectiveness. The WHO suggests that Internet-based prevention interventions should be disseminated after establishing their efficacy [
We regard the assumptions underlying the cost-effectiveness scenarios as conservative, but at the same time it depends on the kind of marketing, that the potential users find the interventions acceptable, and can trust the service provider. Internet-based interventions are costly to develop and require long-term research to test and evaluate their efficacy and effectiveness before they can be disseminated en masse. Locating funding sources for dissemination is often difficult [
Internet-based treatments are more accessible than therapist-delivered treatments and, therefore, have the potential to reach more people with an unmet need [
Although the current study demonstrates that the translation of interventions can be cost-effective, there are other issues that need to be addressed in establishing a translation agreement. Firstly, the licensing deed or contract will need to specify the degree to which the translator can adapt and customize the translated version to local requirements. Secondly, this agreement will need to specify the extent to which the translating site will have the opportunity to implement new functionalities, such as customized printouts. Thirdly, the agreement will need to specify the extent to which the translating site will have an opportunity to influence and contribute to new content for the program. If the content of the application is altered, the application is no longer the same. However, if the application is run by the original center, as was the case for MoodGYM and BluePages, the proliferation of different versions represents a logistical challenge, particularly if programs, such as the BluePages depression information website, requires updating as new information develops, or changes are introduced on the basis of user feedback. Fourthly, the contract should be explicit about the ownership of data collected from the translator’s national users and trial. Finally, special attention should be paid to the timely updating of the translated version after changes in the original one.
There are several limitations of this study. The validation part of this paper is based on mixed model repeated measures (MMRM) to handle missing data, which might have introduced some bias. However, the results used in this report were from the complete case analyses as we needed pretest and posttest for all cases to carry out the estimates for QALYs. The effect of the intention-to-treat sample was somewhat smaller than for the complete cases, and our choice of sample could have strengthened the effect and QOL calculations. The deterioration in the control group is not unusual over a short time [
The economic analysis shows that the cost-effectiveness of the translation project was substantial. This is a natural consequence if the project is proven effective in reducing depressive symptoms and based on unguided self-help. The development cost is fixed and the maintenance cost is minimal. Further, as Internet-based treatments are more accessible than therapist-delivered treatments, when these Web-based interventions are disseminated at a national level, they should reach a significant number of individuals with an unmet need for help. The current findings should encourage others to undertake similar translation activities and facilitate global dissemination of effective programs to reduce health disparities across countries and cultures. When data on societal benefits are taken into account, Internet-based prevention interventions should become even more appealing.
Australian National University
cognitive behavior therapy
cost-effectiveness ratio
Center for Epidemiologic Studies Depression Scale
Cohen’s d
Hedges’ g
degree of improvement in health
degree of improvement in health, extrapolated
International Society for Research on Internet Interventions
Kessler Psychological Distress Scale
mixed model repeated measures
National Institute for Clinical Excellence
Norwegian kroner
quality-adjusted life year
QALYs gained over a timeframe of 1 year
quality of life
randomized controlled trial
World Health Organization
Dr Griffiths is one of the authors and developers of the MoodGYM and BluePages websites, but derives neither personal nor financial benefit from the programs.