Published on 05.03.13 in Vol 15, No 3 (2013): March
Recruitment to Online Therapies for Depression: Pilot Cluster Randomized Controlled Trial
Background: Raising awareness of online cognitive behavioral therapy (CBT) could benefit many people with depression, but we do not know how purchasing online advertising compares to placing free links from relevant local websites in increasing uptake.
Objective: To pilot a cluster randomized controlled trial (RCT) comparing purchase of Google AdWords with placing free website links in raising awareness of online CBT resources for depression in order to better understand research design issues.
Methods: We compared two online interventions with a control without intervention. The pilot RCT had 4 arms, each with 4 British postcode areas: (A) geographically targeted AdWords, (B) adverts placed on local websites by contacting website owners and requesting links be added, (C) both interventions, (D) control. Participants were directed to our research project website linking to two freely available online CBT resource sites (Moodgym and Living Life To The Full (LLTTF)) and two other depression support sites. We used data from (1) AdWords, (2) Google Analytics for our project website and for LLTTF, and (3) research project website. We compared two outcomes: (1) numbers with depression accessing the research project website, and then chose an onward link to one of the two CBT websites, and (2) numbers registering with LLTTF. We documented costs, and explored intervention and assessment methods to make general recommendations to inform researchers aiming to use similar methodologies in future studies.
Results: Trying to place local website links appeared much less cost effective than AdWords and although may prove useful for service delivery, was not worth pursuing in the context of the current study design. Our AdWords intervention was effective in recruiting people to the project website but our location targeting “leaked” and was not as geographically specific as claimed. The impact on online CBT was also diluted by offering participants other choices of destinations. Measuring the impact on LLTTF use was difficult as the total number using LLTTF was less than 5% of all users and record linkage across websites was impossible. Confounding activity may have resulted in some increase in registrations in the control arm.
Conclusions: Practitioners should consider online advertising to increase uptake of online therapy but need to check its additional value. A cluster RCT using location targeted adverts is feasible and this research design provides the best evidence of cost-effectiveness. Although our British pilot study is limited to online CBT for depression, a cluster RCT with similar design would be appropriate for other online treatments and countries and our recommendations may apply. They include ways of dealing with possible contamination (buffer zones and AdWords techniques), confounding factors (large number of clusters), advertising dose (in proportion to total number of users), record linkage (landing within target website), and length of study (4-6 months).
Trial Registration: clinicaltrials.gov (Registration No. NCT01469689); http://clinicaltrials.gov/ct2/show/NCT01469689 (Archived by WebCite at http://www.webcitation.org/6EtTthDOp)
J Med Internet Res 2013;15(3):e45
- cluster randomized trial;
- pilot study;
- online advertising;
- Living Life to the Full, LLTTF;
- Google Analytics;
- computerised CBT
Less than 60% of people with diagnosable depression or anxiety seek formal help from practitioners; this represents a significant treatment gap . The remainder may access informal care, alternative therapies, make private arrangements such as counselling, use the voluntary sector, the Internet, or use other sources of information. Nearly 1 in 5 British Internet users search for information related to mental health [ ], but patients searching for health information online may not find what they are looking for [ ], possibly due to sub-optimal search strategies [ ].
There is increasing evidence that online interventions can be effective in changing health behaviors or improving health . Online cognitive behavioral therapy (CBT), is an example offering effective self-help treatment for depression. Online CBT is recommended by the National Institute for Health and Clinical Excellence (NICE) in the UK for mild to moderate depression [ ]. The range of resources includes licenced (paid for) sites such as Beating the Blues [ ], and free access websites providing access to CBT life skills resources (eg, Living Life To The Full (LLTTF) [ ] and MoodGYM [ ]).
However, many patients do not benefit through lack of awareness. We found a variation in registration to LLTTF of 15-fold between the highest (Kirkwall, Scotland) and lowest (Wigan, England) postcode areas . Variations in Internet use by region in Britain are small; 80.3% of those in the North East had at some time accessed the Internet compared to 87.6% in London by 2012 [ ]. Variation in the prevalence of depression between postcode areas is small [ ], and although other packages such as MoodGym may be in use, the most likely explanation for variation in LLTTF registration was lack of awareness.
Raising awareness of online CBT could benefit many people with depression by facilitating early rapid access. Ways of addressing this include online advertising through search engines such as Google AdWords (AdWords) [- ], advertising on social media sites [ ], “snowballing” on social media sites [ ], getting websites of other organizations to add links (weblinks) [ ], using offline via mass media [ ], or via practitioners [ ].
AdWords have been used by others to recruit participants to studies, for example, for depression screening , use of condoms [ ], and quit smoking campaigns [ ], but the cost effectiveness of their use has not been assessed. Weblinks from a range of existing sites have also been used in research studies, often routinely as part of a “recruitment package” (eg, [ ] and LLTTF is linked from sites such as the English National Health Service’s (NHS) website NHS Choices [ ]), but we were unaware of any study of weblinks restricted to local organization websites in order to target recruitment on a specific local population as required for this design.
Studies that compare recruitment methods before and after interventions have no control group that could be described and compared with adequately. The only rigorous way of comparing methods of raising awareness of online therapies is by geolocated cluster randomised controlled trials (RCT, ). Matching intervention and control areas reduces the chances of bias. If we can limit online advertising to one geographic area and compare it with another area where there is no advertising, we can then assume that any difference is due to the advertising. We can therefore estimate its cost effectiveness in terms of cost per new user. Subsequently, we can decide if it is worth using online advertising to raise awareness, or whether other methods would be more cost effective.
We planned to compare purchasing AdWords with the second strategy of using weblinks and carried out a pilot RCT of the two different recruitment interventions to check the methods and outcome measures for a definitive trial later. In particular, our study objectives were to explore: (1) whether or not the two recruitment interventions seemed to work at all, and so be worthy of further study, (2) the ability to target online Google adverts or weblinks from other local websites without contamination, (3) the ability to link data sources to effectively measure the impact of the recruitment interventions, and (4) to learn more about the likely size and impact on target CBT site use and, given the other methodological issues such as confounding factors, to know what sample size and dose of advertising will be needed for future substantive studies.
Ethics and Registration
The study was approved by the NHS South West 2 Research Ethics Committee (Reference 11/H0203/8; February 2011) and registered on clinicaltrials.gov (Registration No. NCT01469689).
This was a pilot cluster RCT of recruitment interventions for online CBT for depression. We compared two online recruitment interventions with a control without intervention. The pilot RCT had 4 arms: (A) geographically targeted paid AdWords, (B) free adverts/weblinks placed on local websites by contacting those website owners (weblinks), (C) both interventions combined, (D) control. Participants were directed to a project recruitment website linking to two freely available online CBT sites (Moodgym and LLTTF) and two other sites (Samaritans  and NHS Choices [ ]). We used data from (1) AdWords, (2) Google Analytics (Analytics) for our project website and for LLTTF, and (3) our Online Help For Depression (OHFD) project website. We examined 2 outcomes in the 4 arms of the study: (1) numbers accessing the project recruitment website that had depression, and then chose an onward link to one of the two CBT websites, and (2) numbers registering with LLTTF. We documented costs and explored intervention and assessment methods to make recommendations for a definitive trial, for other researchers carrying out similar studies, and tentative recommendations for practitioners and policy makers. shows a schematic of the design.
Sampling and Randomization
All 121 postcode areas in England, Wales, and Scotland were divided into quartiles by rate of registration on LLTTF (based on 36,753 registrations of people between June 2008 and June 2009 who scored 8 or more on the Hospital Anxiety and Depression Scale for either anxiety or depression) and into quartiles by population size (based on the 2001 census). We randomly selected 4 cells from this array, and chose 4 nearly consecutive postcode areas for each cell (to try to achieve similar populations), avoiding adjacent geographical areas. Each set of 4 postcode areas was randomly allocated to the 4 arms of the trial (three interventions and control). Randomization was not blind.
Each arm of the study had a total population ranging from 1.6 to 2 million people clustered in 4 postcode areas. In total the study included 7 million people in 16 postcode areas across England, Wales, and Scotland. The estimated point prevalence for major depression among 16- to 65-year olds in the UK was 2.1%, rising to 9.8% when the less specific and broader category of “mixed depression & anxiety” was included . We used an intermediate estimate of the prevalence of depression of 5%, which gave an estimated target population of 350,000.
Those paying for AdWords campaigns set up one or more adverts, and enter keywords to help determine when the advert is shown. AdWords displays adverts as a sponsored link, either at the top of the list of search results or in the right hand search results panel, depending on the phrase entered, the price offered per advert, bids from competing adverts, and (if requested) by estimated location of the user.
Within the 8 areas in arms A and C, we ran AdWords from April 17th to November 30th, 2011. In the 8 areas in B and C, we aimed to place adverts (weblinks) from local organization websites such as local universities, general practitioner (GP) practices, and local authorities, by contacting these organizations via email and/or phone starting on April 17th. Arm D was a control arm with no recruitment intervention.
We used a single advert () from April 17th to October 19th as one campaign with a daily budget of £7.50 per day (4.4 pence per 1000 target population per day). Targeting specific postcode areas (eg, Kingston, KT) was not an option offered by AdWords. Options did however include targeting a radius of 1 mile or more around a postcode district (eg, KT2) or to hand draw a polygon to enclose the area of interest. We used a mix of methods: 4 postcode areas were defined using circles of 1 mile radius for all postcode districts within the postcode area and 4 were defined by hand drawn polygons. Preliminary analysis [ ] showed leakage from the target areas and an imbalance in the number of presentations of the advert between postcode areas. This prompted a change of strategy with 8 separate adverts for each postcode area and adverts that mentioned the target area. Radius targeting (rather than polygon targeting) was used for all areas from October 19th to November 30th. The daily budget was increased to £16.10 (9.3 pence per 1000 target population per day) and divided between the 8 areas in proportion to target population.
We originally asked AdWords to display the advert for the keyword of depression. AdWords suggested other similar keyword combinations and we accepted all suggestions. AdWords gave information on the number of times they presented adverts against Google searches, by day, keyword, and location. AdWords decided when to present the advert based on the price we offered, the price of competing adverts, and other factors such as the search terms used. Users searching on terms such as depression and depression help were presented with our advert, depending on our budget and competing adverts.
Google searches were used to identify organizations in 8 postcode areas (arms B and C) with websites to place adverts (weblinks to our project website) on local free access websites. In our search we included websites such as local GP surgeries, local media websites including newspaper, TV, radio, further and higher education institutions, and community-based or local charity websites. In total, 180 emails were sent with 3 weblinks posted free of charge: two in Leeds (university medical practice and a carers’ organization), and one in Kirkwall (local online community newspaper).
Those who clicked on Google adverts or on weblinks were directed to our project research website OHFD. This gave information about the study and advised that completion of the online questions implied consent for it to be used in the study (seefor screenshots). Computer Internet Protocol (IP) addresses, dates, and times were collected via OHFD, and also monitored by Analytics. We specifically did not try to raise the visibility of our website to normal Google, Yahoo, or Bing search engines. Visitors were asked for their postcode area and to complete the Patient Health Questionnaire (PHQ9) [ ] assessing depression. Users were then offered 4 links to Moodgym, LLTTF, NHS Choices information on depression, and Samaritans. The order in which the links to Moodgym and LLTTF (top row) and NHS Choices and Samaritans (bottom row) appeared was randomized within row.
We used various sources of data to model patient flow (). AdWords and Analytics were used for OHFD. The website log for OHFD recorded stated postcode areas from participants and their website destination choice (if made). Analytics was used for all visitors of LLTTF, identifying those referred by OHFD and those likely to be in study arms (using Google defined locations). Website log was used for LLTTF, including those who registered and stated their postcode area.
We documented costs for using AdWords and other weblinks and estimated costs per person with depression referred to the online CBT via AdWords, based on time spent, cost £10/hour (based on the hourly rate Plymouth University pays temporary administrative staff) and Google’s charges. We included costs needed for routine delivery of these methods and excluded research costs such as time spent in the comparison of methods or in setting up the research project website.
We defined 2 main outcomes and compared them between the 4 arms: (1) the numbers accessing the project research website (OHFD) that completed a PHQ9 depression rating score and had a score of more than 5, indicating at least mild depression, and then chose an onward link to one of the two CBT websites, and (2) the numbers registering with LLTTF who gave their postcode, comparing intervention with control and all areas. In the revised version of LLTTF (issued in January 2011) used in this study, new arrivals at LLTTF do not have to immediately register, instead, registration can be delayed. Registration, by giving more personal details and agreeing to email reminders, signifies a commitment to use the site seriously. In the previous version of LLTTF used to select the sample, most visitors to the site registered as they could not access most content until after registration.
We carried out this study at a time when the Improved Access to Psychological Therapies (IAPT) project  was investing large amounts of money to improve access to psychological therapies across England. To try to identify other local initiatives and how these may have affected our sample and interventions, we aimed to recruit a panel of informants in our study areas. We emailed five ex-users of LLTTF from each of the 16 areas (ie, total 80), and also tried to contact them via IAPT teams. Panel members were to be paid £30 at the end of the study by e-vouchers.
Did the Two Interventions Work at All?
Examining the number of people clicking through from online adverts to OHFD to online CBT (), we saw that our OHFD research site had a total of 8231 visits, the majority (97%, 7980/8231) from Google adverts and only 1% (103/8231) from weblinks. Half of those who visited OHFD (50%, 4118/8231) interacted with the site and 76% (3135/4118) made a choice of destination from the 4 available sites, three-quarters of whom (77%, 2403/3135) chose the online CBT. Although 96% (2306/2403) had depression (according to the PHQ9 score >5), only 515/2306 (22%) were from the study target areas ( ).
shows that the rates of referral to online CBT for AdWords arms A and C were higher than the weblinks only arm B (450 and 387 vs 93, respectively, per 100,000) and much higher than the control arm (D) (28 per 100,000). These results were confirmed by fitting a Poisson generalized linear model with the R software [ ], with predictors for the use of AdWords, weblinks, and an interaction term for both. This produced a 95% confidence interval for rates relative to control of (10.2 and 27.4, respectively) when using Google adverts. The rate relative to control for weblinks was (2.0 and 6.0, respectively) and there was little to be gained from using both, given an interaction between AdWords and weblinks with rate relative to baseline of (0.1, 0.5). The majority of those choosing online CBT chose LLTTF (66%, 1581/2403) and the difference between arms A/C and B/D was still evident ( ).
Was it Possible to Link Data to Measure Outcome Two?
It proved impossible to directly track individuals from OHFD to LLTTF using IP address and time data, therefore the impact of the interventions on LLTTF had to be assessed using Analytics and LLTTF website data. Analytics reported 1474 visits landing on LLTTF from OHFD, agreeing approximately with the 1581 referred by OHFD, although the 7% attrition is unexplained (). During the period of study, according to Analytics, the majority of the 230,441 visits to LLTTF were from normal search (41%, 93,983/230,441) or direct (37%, 85,000/230,441), with 22% (51,406/230,441) from referring sites. In total, there were 1888 sites from which there were 51,406 visits. Of the referring sites, OHFD was the seventh largest referrer with 1474 but representing just 2.9% (1474/51,406) of referrals and less than 1% (1474/230,441) of all visits. The Royal College of Psychiatrists sent most referrals (5071/51,406, 9.9%), but this still represented only 2.2% (5071/230,441) of all visits.
We know from Analytics that the bounce rate (ie, those who exited from the first page) for visits from OHFD was 47% (no different to the average bounce rate from referred visits of 49%). This suggests that those arriving from OHFD were not more or less likely to continue and to subsequently register. We can see from the difference between arrivals on the site (14,396 from Analytics) and registrations (1143 from LLTTF log data) from the study arms that 10% (1067/10569) of all visitors registered on the current version of LLTTF.
shows that the 95% confidence intervals of rate of registration on LLTTF (outcome 2) overlap between all 4 arms, in other words there was no significant difference between the interventions or control. Although none of the differences for individual postcode areas was significant, registration rates for most (11/16, 69%) postcodes tended to decline from January-April to April-November. Of the more populous postcode areas, Liverpool (arm A) showed the greatest increase (360 to 489, 26% increase) but this was still not significant and Nottingham (in the control arm D) had a 6% increase. Two of the small Scottish island postcode areas may have increased but their populations are small, the confidence intervals on estimates are large, and the impact on the whole arm small.
People on OHFD with depression choosing online CBT and LLTTF
People registering on LLTTF before the intervention (Jan-Apr 2011) and during the intervention (Apr-Nov 2011)
|Estimate of people with |
|People with |
depression who chose online CBT
|Annual rate per 100,000 |
|People with depression who chose LLTTF||Rega|
|Annual rate per 100,000 depressed|
|Annual rate per 100,000 depressed|
|E, W, and S||28,54373||2306||129|
b1581 includes 87 who gave no postcode on OHFD, but all other indications (eg, IP address) show that they were England (E), Wales (W), or Scotland (S).
Validity Check on Location
As a validity check on location, we carried out an alternative analysis on those registering on LLTTF using the Analytics location data instead of user stated postcodes. The two approaches showed agreement.
Possible Confounding Factors
According to Analytics, the biggest increase in “landings” on LLTTF was seen in one of the control areas, Nottingham.(middle panel) suggests that there may have been a “step change” for Nottingham in October 2011. Overall, the numbers visiting LLTTF were largely unchanged over the study period (apart from the very regular weekly cycle of visits with fewer during weekends, bottom panel).
Size of the Impact on LLTTF
Ignoring the stated locations of those clicking through to LLTTF, we see that 1581 clicked through to LLTTF in the study period (). In the study period, the 3 intervention arms had 10,569 visits (estimated from Analytics) but only 1067 people registered, that is, with the current method of counting registration, only 10% of visitors to LLTTF registered. So if we estimated that, overall,10% of those referred from OHFD registered, we can see that the numbers referred from OHFD (for example, 17 in Arm A, ) were likely too small to greatly influence the total numbers registering in each of the trial arms ( , outcome 2-people registering on LLTTF). As we were unable to link individual participants from the project website (OHFD) to LLTTF, we could only tell if there had been an impact on the number of people registering on LLTTF if it was of sufficient size. If it had been possible to follow through individual participants from clicking on an advert to registering on LLTTF, having an impact on the total number registering from each area would have been less important.
Just under 8000 (7980/8231, 97%,) visits to OHFD came from Google adverts, of which 4118/7980 (52%) interacted with OHFD, 3135/4118 made a destination choice, but only 515/3135 (16%) were from study arms with PHQ9 >5 who chose either Moodgym or LLTTF. Of these, only 387/4118 (9.3% of those who interacted with OHFD) chose LLTTF. Of those choosing LLTTF, probably only 10% registered, ie, less than 0.5% of those who clicked on the advert were likely to have registered on LLTTF. The biggest sources of “leakage” were from the location targeting [ ] and from participants not engaging fully on LLTTF. The leakage however did not lead to much contamination of control areas. Failing to continue with the OHFD website and being lost to other or no Web destination also caused substantial leakage.
All 3 weblinks were for locations in arm B. Arm C therefore was effectively another AdWords arm. The total cost of the AdWords campaign in payments to Google was £1841. From April 17th to October 19th, we set a daily budget of £7.50 (4.4 pence per 1000 population) and had one campaign in which all 8 areas were included. As we described elsewhere , there was a disproportionate spending on AdWords for London SW. To counter this and to better understand the responses in each area, on October 19th, a new campaign in which each postcode area had its own budget was started. For the remaining 6 weeks, we increased the daily budget to £16.10 (9.4 pence per 1000 population), and divided this in proportion to the target population in each area (but giving a minimum budget of 60 pence for Shetland). shows that the average number of clicks per day in the last 60 days was 29.3 (1759/60) compared to 34.2 (6291/184) in the first 184 days. This method (restricting the spending on London SW) also resulted in a higher cost/click overall, although we had noted in the first period that it takes AdWords 3-4 weeks to gain the optimum return on advertising spending after starting a new campaign.
We estimated that the time spent simply adjusting the AdWords campaign, as opposed to time spent on Analytics trying to match with other data, was 25 hours. At £10/hour, the total cost of the AdWords campaign was £2091 (£1841+£250). If we assume that both arms A and C were AdWords, this represents £4.64 per person for the 451 people who chose online CBT and had a PHQ9 >5 in arms A and C (the target group,). Seventy hours were spent trying to contact owners of local websites to set weblinks, which if set at £10/hour, gives a total cost of £700 for arm B. This represents £14.89 per person for the 47 people in arm B ( ).
|Area||Maximum daily |
|Average daily spend||Clicks||Cost||Cost per click|
|April 17th-October 19th(184 days) 2011 single campaign|
|8 individual campaigns by area October 20th-November 30th (60 days)|
In total, we were able to recruit 12 panel members and all arms, but not all postcode areas, of the study were represented by either an ex-user of LLTTF or a member of an IAPT team. No other local interventions were identified via the user panel but the panel was able to help in examining the search environment across Britain .
Need for the Study
McCrone estimated the number of people with depression in England as 1.24 million (2.3%) with total cost of services at £1.7 billion while lost employment increased this total to £7.5 billion . Others estimated that a third of people with depression are not in contact with services [ ]. Online CBT has a significantly small-to-medium effect size compared to non-active controls for patients with a range of severity of depressive symptoms and was recommended by NICE for mild to moderate depression [ ]. There is virtually no extra cost in additional people using online CBT websites such as LLTTF but there was evidence that many people with depression who might benefit from using online CBT were missing the opportunity of effective treatment simply through lack of awareness. By raising awareness of its availability directly to depressed people, we could reduce inequalities in access to this treatment.
Various ways of raising awareness of websites are available including online advertising and weblinks. We did not know if online adverts such as AdWords would be more or less effective than weblinks. These methods may simply attract the same people that would in any case have found online CBT. As a pilot study, we were looking at the feasibility of being able to answer these questions and what study design would enable us to do so.
Which Interventions Seemed to Work?
We have found that using AdWords is possibly going to be cost effective, whereas trying to get weblinks is not worth pursuing further in this locality based research study design. We only managed to put 3 weblinks in place. Scaling up the AdWords campaign would incur further advertising but no further labour costs (so the unit price gradually decreases), whereas scaling up the weblinks campaign would be proportional to the labour costs. Establishing weblinks therefore appears much less cost effective than AdWords and is not worth pursuing as a sole intervention in this study design.
However, the ability to establish weblinks may be significantly easier for local organizations trying to offer local services. In our study, we chose areas remote from the research team, and there were no prior local relationships on which to build. Being locally situated and delivering local services to local people might significantly affect the ease of establishing local websites. Others have used weblinks to successfully recruit as part of their recruitment package. For example, 91% (174/191) of participants in a trial of online treatment for chronic headache learned of the study from weblinks (both mutual and paid for) on websites, registration with major search engines, and notices posted to headache-related news groups, but the authors did not compare these methods and the study was global . Overall 1 in 5 come to LLTTF from a weblink and various national website including NHS Choices and Royal College of Psychiatrists already have links to online CBT [ ] and other nationwide charity websites may be willing to include weblinks. However, pursuing local websites such as primary care health centres to fit with this geo-targeted research design did not seem cost-effective. Nevertheless, this should not be discounted as an option for practitioners where it is known that weblinks can be put in place with little effort.
Design Issues to be Addressed in a Definitive Trial
AdWords were effective in recruiting people to the project website but our pilot study identified a number of issues for our, and similar, cluster RCTs. The main problems were leakage so that its effect was greatly diluted, problems in linking data across websites so that the impact of advertising was lost in the large numbers using LLTTF, and possible confounding factors. The dose of advertising (particularly given the “record linkage” problem) and length of study also need to be considered. The changes that would be needed to the design of a definitive trial are discussed below. These findings will be relevant to others seeking to improve the uptake of online interventions or designing online cluster RCTs.
There was leakage at 3 points in the process. First, we have shown elsewhere  that there was leakage from the target areas, particularly into neighbouring areas. This was partly due to the methods used and could be improved by using an area that is within the radius of set distance around a point rather than hand-drawn polygons, use of more appropriate radius, and avoidance of the area “edge”. Second, there was leakage caused by offering too many choices of destination to participants. We had included both MoodGym and LLTTF as the two main free online CBT sites as it was thought that there might be geographic variation in choice of online CBT. For ethical reasons we added Samaritans as another option, and added NHS Choices to create a balanced (even) number of options. As a result, only 50% of those making a choice chose LLTTF. As we did not have access to Moodgym log data, we were only able to follow participants to the point of choosing online CBT, and the impact on one of those sites (LLTTF). Third, with the current system of registration on LLTTF, only 10% of those accessing LLTTF decided to register and enter demographic data including their postcode (LLTTF allowed people to use the website without registering; registering offered further features such as email reminders).
To decide whether a definitive trial will be feasible, we modelled the effect of reduced leakage. By taking 50% of the number lost on the project website before making a destination choice, by routing all participants to LLTTF, and by reducing losses from location targeting from 76% to 20%, we would reduce the leakage from 94% to 47%.
Record Linkage and Insufficient Dose
We were not able to track individuals from OHFD to LLTTF and therefore could only use the overall data to try to estimate outcome 2, the number of people registering with LLTTF. Although outcome 1 showed significant differences between intervention and controls, to be able to measure a difference, the number referred from AdWords needed to be sufficiently larger compared to the total number that registered on LLTTF. The number of people with depression referred from OHFD to LLTTF was 4.7% (498/10569) of all users of LLTTF in study arms. If, as above, leakage was reduced to 47% the number referred by OHFD would still only be 10% of all those landing on LLTTF. To be certain that an impact of AdWords can be seen, the dose of advertising also needs to be increased. Although we doubled the daily budget to 9.3 pence per 1000 target population for the last two months of this pilot study, the number of clicks was slightly less than in the first period, probably due to breaking the advertising into 8 separate campaigns and AdWords not having had time “to settle”. The effectiveness of the AdWords advertising budget depends on competing adverts, and is likely to be less cost effective as the daily budget is increased. As shown in, the average daily spend was short of the maximum budget so it is not clear if an increase in daily budget would be spent or would increase the number of clicks, but a further doubling of the advertising budget would be worth trying.
A major contributing factor to the lack of impact on outcome 2 was the increase in uptake of LLTTF in one of the control areas (Nottingham), possibly as a result of a presentation given to practitioners by one of the authors (CW) to a national conference that overlapped with a surge of use. Although it was not a blinded region selection, day-to-day management of the study was conducted by RJ. We recognized in designing the pilot study that random interventions or effects from other influences, such as local campaigns, could impact on intervention or control areas. For that reason, we had attempted to monitor all areas via the user panel. This pilot study suggested that, to avoid an overdue influence of one postcode area (cluster), each arm needs many more than 4 clusters. As we had designed the pilot study with 2 types of interventions and 4 arms, we only used 16 out of 120 postcode areas for our study.
A more robust approach would be to use all postcode areas in Britain, excluding London postcodes, ordered by population size only, randomized in pairs to two arms, excluding any adjacent postcode areas as buffer zones around study areas. Using this method produces a two-arm study with 32 postcode areas (see) and population of approximately 10 million in each arm. One arm would then be randomized to the intervention and one to the control group. The largest postcode area in one arm (Birmingham) still represents 17% of the total arm, so there is still some danger that other activity might lead to confounding factors, however this is much reduced from the design we used in this pilot where Nottingham represented 55% of the control arm population. This sample ( ) of postcode areas could be adopted for other studies in Britain.
Other Changes to Improve the Design of the Definitive Trial
Length of the Study
We paid for over 8 months of advertising but the recruitment numbers stabilized after a few months and we did not change the advert for 6 of the 8 months. On the other hand, it takes 2-3 months for AdWords to reach peak efficiency. With reduced leakage, a bigger sample, and an increased daily budget on advertising, the cost of a definitive study could probably be reduced by examining changes over 6 months.
Need for Better Methods of Assessing Location
Assessing location is subject to error. Previously , we have given a detailed comparison of user-reported versus methods based on IP address including using IP lookup tables and Analytics. The method we used to ask participants for their postcode area on our project website (OHFD) was probably the most accurate. We had hoped to be able to use IP addresses and the time of referral from OHFD to LLTTF to track individuals and so to estimate rates of completion of these “additional” registrants with other registrants. However, the changes to LLTTF registration procedures and the difficulties of trying to match IPs from one website to another using time of day made this impractical. LLTTF currently ask participants for their full postcode which probably deters some from giving any information and may encourage falsification or error. We have recommended changes to data collection methods in LLTTF, asking for postcode area from a drop down list. Analytics location information would be more useful if the area associated with a town name was transparent to users, and if it could be aligned with population figures. We have tried to suggest to Google that their geography should be changed. In the meantime, the geography used by Google for London and immediate surroundings suggest that, to have more accurate location data in a study, London should be excluded.
Finding a Period of Website Stability
Studies that aim to change Internet use will be limited by frequent changes to websites and other technical advances. In our case, even though CW was the author of LLTTF, changes underway to the LLTTF website could not be postponed for this study. Our sample was selected based on data extracted from LLTTF between the years 2008 and 2009 when most users of the site registered. In January 2011, after our study had been designed and ethical approval had been sought, LLTTF was reconfigured such that registration was optional and could be done at a later time. As a result, the number of people that registered greatly reduced. This meant that registration figures were no longer directly comparable with those collected earlier. Although we were able to compare the relative differences between regions in 2009 with 2011, by creating an index of use based on the lowest use region, we restricted direct comparison (ie, the “before” period) to January- April 2011.
Graham  found that online advertising could be an effective and cost-efficient strategy to reach and engage Spanish-speaking Latino smokers in an evidence-based Internet cessation program. She concluded that cultural targeting and smoking-relevant images might be important factors for banner advertisement design. In this pilot we did not collect demographic details. These were collected by the target LLTTF website as part of registration and we hope to achieve better record linkage in the future.
In this pilot study we selected our sample trying to match for the populations of postcode areas and the baseline registration rates on LLTTF. Although it would be important not to have a grossly imbalanced sample, the need for a greater number of postcode areas (as described above) while ensuring geographical dispersion to avoid contamination would seem to be more important in the design.
Comparison Against Other Methods
Although various online methods exist to raise awareness, the problem of being able to select sample areas allowing for the competing demands of contamination and confounding factors suggest that having a factorial design with more than one recruitment intervention would be difficult. On the other hand, for practitioners seeking to increase recruitment using a mix of all possible methods, more than one recruitment intervention would seem to be sensible.
Conclusions for Practitioners and Policy Makers
Our pilot study confirms other research that many people search online for help with mental health issues  such as depression; our advert was displayed 673,074 times in just over 7 months for a total targeted population of 3.5 million. Anecdotally, in discussing this research, many practitioners responded by saying “I never click on adverts” but our pilot has shown that nearly 8000 people clicked on our short advert for NHS recommended sites and many of these were depressed, according to their answers on a self-completed questionnaire.
In discussing this research with NHS policy makers, the response to advertising, perhaps in the light of previous criticisms of their expenditure, was that NHS websites have been optimized and therefore appear high in search results, so there is no need for online advertising. This may be true but our exploration of this  suggested that even though online CBT sites can eventually be found via NHS and Royal College of Psychiatrists websites that tended to appear high in search results, the probability was low, and was significantly increased for a naïve user by the addition of an advert. That evidence however was theoretical and the only way of knowing for sure about the cost-effectiveness of online advertising was to study it in a location targeted RCT.
Although the results of the weblinks in this pilot study was not as expected and will be excluded from this research design (unless a better method of placing local weblinks can be found), it would be wrong to conclude that weblinks are not relevant for practitioners and policy makers. The problem may be that the type of site containing relevant links tends to be either national (inappropriate for a location targeted clusters RCT) or very local (researchers likely do not have prior knowledge about existence of these sites). Practitioners and policy makers may therefore have to rely on weaker evidence from before/after studies of cost and impact to decide on how much effort they put into using weblinks.
Part of the reason why LLTTF had greater use in Scotland compared to other parts of Britain is that professionals in those areas were more aware of the existence of these sites and recommended use of these sites to patients. The role of professionals in recommending online CBT may also have explained the confounding results seen in one area in the control group. So it may be that continuing professional development and raising awareness among professionals about resources for depression may be as, or more effective, than direct-to-patient online interventions. However only one third of patients were in contact with health services and other studies have shown that trying to reach patients to tell them about online resources may be labour intensive, time consuming, and very expensive . Further research is needed to compare the cost-effectiveness of improving access via professionals versus direct to patient methods, but in the meantime, practitioners and policy makers should keep online advertising as an option.
Conclusions for Researchers
A cluster geo-located RCT to test the cost-effectiveness of online advertising seems feasible in Britain. Geolocated adverts are offered by Google in other countries based on a radius around a point, so the general design of a cluster RCT to test the cost effectiveness of online advertising in raising awareness of an online therapy, as piloted in this study, would seem to hold true for other countries and for other online therapies or websites. However, this pilot study has demonstrated 4 general messages concerning contamination, confounding, dose of advertising, and length of study that will be useful for other researchers.
Provided it is possible to have a sufficiently large buffer zone between intervention and control regions, it should be possible to deal with potential contamination. The definition of sufficiently large is vague and needs to be piloted in each country, but in Britain, we think a 2-arm trial with 16 postcode areas in each should be possible without too much contamination. The problems of including London in Britain were described in more detail elsewhere . It seems likely that similar studies in other countries may need to exclude the capital or major centres for Internet providers. We have also previously described other design issues with using AdWords including the need for separate campaigns for each postcode area [ ].
Ideally, such a trial would have a large number of clusters in each arm such that any one region is kept to approximately 5% of the total population in the arm. That would mean that any confounding activity such as local media campaigns would not influence the arm greatly. However, dealing with confounding factors conflicts, to some degree, with the relatively small and densely populated country like Britain. In our proposed best design for Britain, trying to keep contamination to a minimum at the postcode area level, our design of 2-arms of 16 postcode areas each, still included a city representing 17% of that arm. This was slightly risky for introducing confounding factors but was the best we could do. In a bigger country such as the United States, it should be possible to have a stronger design. In a smaller, densely populated country such as the Netherlands, this study design may be impossible.
Dose of Advertising
We recommend that in such a design, adverts are linked to a special “landing page” within the target site. This allows those following the link to track their use of the target site. We had used an intermediate project website in this pilot because we were offering 2 online CBT sites and 2 other sites. This gave us considerable problems in trying to track participants from the advert to the project website to the target website, and in trying to match up the different data sets available from AdWords, Analytics, and website logs. Having a landing page within the target website should remove most problems but it would still help in the design if the dose of advertising, and so the expected “footfall” from advertising was substantial compared to the number arriving from other sources. In this pilot, with a limited advertising budget, although we had significant numbers arriving at our project website, the problem of leakage meant that the number of people finding the target website compared to the numbers finding it from a normal search, national weblinks, or direct entry were small. Any study with this type of design should pay greater attention to the total footfall on the target website and try to calculate the dose of advertising needed to have a demonstrable impact.
Length of Study
The cost of advertising in such a trial will be partly determined by how long the campaign is run. We ran our pilot for 7.5 months. Although it did take several weeks for our campaign to stabilize and for AdWords to get the best return and cost per click, a campaign of 4-6 months should be sufficient for this type of study where the total number clicking is in the range of 300-500 people. The best design (as described above) is to have a larger dose of advertising over a shorter period rather than a small dose over a longer period.
We cannot be sure how the findings of this study would translate to other countries and there is no guarantee that location targeting of online advertising will continue to be available in this form. Increasing use of mobile phones may change the way location targeted adverts work. AdWords is of course not the only way of local advertising online and a definitive trial might consider use of advertising solutions from Microsoft, Facebook, LinkedIn, and others. Facebook, for example, offers location targeting and would also be worth exploring in this way.
Our study was limited by the difficulties of trying to match different data sources. Different sources from Google (AdWords and Analytics) do not exactly match due to different ways of collecting the data. Other issues, because of the anonymity of the data, include whether visitors are unique individuals. For example, although Analytics may claim to report unique visitors, we cannot verify that claim and it would be impossible for Analytics to differentiate between two individuals using the same computer (IP address) and one person with two emails using the same computer. LLTTF only collects emails of those who register. It is likely, therefore, that all sources overestimate the number of unique individuals. However, although numbers from different sources do not match exactly, the overall picture seems consistent and reasonably robust.
This pilot study has shown that a definitive cluster trial of AdWords is worthwhile, and that this type of design could be used to assess other online recruitment interventions.
We would like to thank Ian Mayer of Mirata Ltd for help in extracting anonymised data from LLTTF, Robert Stillwell for developing the project website. RJ developed the idea for the study, was the principal investigator and grant holder of the study. RJ was responsible for day-to-day management of the project, carried out most of the analysis, wrote and edited the paper. LG contributed to the research proposal, was a co-grant holder, undertook day-to-day management, did some of the analysis, helped write and edit the paper. PH reviewed the whole study, in particular the statistical methods in the paper, suggested an alternative method to model the data, carried out that modelling, and edited the paper. CW contributed to the research proposal, was co-grant holder, and edited the paper. The project was funded by a grant from the Bupa Foundation Philip Poole-Wilson Seedcorn Fund. Ethics committee approval reference: 11-H0203-8.
Conflicts of Interest
Chris Williams is the designer and author of the LLTTF site, and is Director of Five Areas Ltd which licences the LLTTF website. LLTTF is run under licence by the charity Jubilee Trust.
Multimedia Appendix 1
Screenshots from Online Help For Depression.PDF File (Adobe PDF File), 314KB
Multimedia Appendix 2
Two arms for definitive cluster randomized trial, showing the 16 postcode areas in each arm, cluster populations, and total population for each arm.PDF File (Adobe PDF File), 11KB
- Kohn R, Saxena S, Levav I, Saraceno B. The treatment gap in mental health care. Bull World Health Organ 2004 Nov;82(11):858-866 [FREE Full text] [Medline]
- Powell J, Clarke A. Internet information-seeking in mental health: population survey. Br J Psychiatry 2006 Sep;189:273-277 [FREE Full text] [CrossRef] [Medline]
- Fox S. Washington, DC: Pew Internet and American Life Project Online Health Search 2006 URL: http://www.pewinternet.org/Reports/2006/Online-Health-Search-2006.aspx [accessed 2012-11-20] [WebCite Cache]
- Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 9;324(7337):573-577 [FREE Full text] [Medline]
- Jones R, Goldsmith L. Plymouth: University of Plymouth; November. 2009. What is the evidence for the benefits and outcomes for digital health services? Final Report to NHS Choices URL: http://www.plymouth.ac.uk/researchcover/rcp.asp?page=256&pagetype=G [accessed 2011-11-04] [WebCite Cache]
- Depression: the treatment and management of depression in adults (update). London: National Institute for Clinical Excellence; 2009. URL: http://publications.nice.org.uk/depression-in-adults-cg90 [accessed 2013-02-22] [WebCite Cache]
- Beating the blues. URL: http://www.beatingtheblues.co.uk/patients/ [accessed 2012-11-20] [WebCite Cache]
- Williams C. www.llttf.com. 2012. Living life to the full URL: http://www.llttf.com/ [accessed 2011-10-31] [WebCite Cache]
- MoodGYM. 2012. URL: https://moodgym.anu.edu.au:443/welcome [accessed 2012-11-20] [WebCite Cache]
- Jones RB, Goldsmith L, Williams CJ, Kamel Boulos MN. Accuracy of geographically targeted internet advertisements on Google AdWords for recruitment in a randomized trial. J Med Internet Res 2012;14(3):e84 [FREE Full text] [CrossRef] [Medline]
- Office for National Statistics. 2012. Internet access quarterly update, 2012 Q2 URL: http://www.ons.gov.uk/ons/publications/re-reference-tables.html?edition=tcm%3A77-268719 [accessed 2012-11-20] [WebCite Cache]
- Weich S, Holt G, Twigg L, Jones K, Lewis G. Geographic variation in the prevalence of common mental disorders in Britain: a multilevel investigation. Am J Epidemiol 2003 Apr 15;157(8):730-737 [FREE Full text] [Medline]
- Estrada C, Krishnamoorthy P, Smith A, Staton L, Houston T. 2009 Southern Regional Meeting Abstracts. Web-analytics: changes of participation measurement URL: http://www.afmr.org/abstracts/2009/SR2009_abstracts/431.cgi [accessed 2012-11-20] [WebCite Cache]
- Lee K, Otgonsuren M, Younoszai Z, Mir HM, Younossi ZM. Association of chronic liver disease with depression: a population-based study. Psychosomatics 2013 Jan;54(1):52-59. [CrossRef] [Medline]
- Estrada CA, Krishnamoorthy P, Smith A, Staton L, Korf MJ, Allison JJ, et al. Marketing to increase participation in a Web-based continuing medical education cultural competence curriculum. J Contin Educ Health Prof 2011;31(1):21-27 [FREE Full text] [CrossRef] [Medline]
- Ramo DE, Hall SM, Prochaska JJ. Reaching young adult smokers through the internet: comparison of three recruitment mechanisms. Nicotine Tob Res 2010 Jul;12(7):768-775 [FREE Full text] [CrossRef] [Medline]
- Graham AL, Bock BC, Cobb NK, Niaura R, Abrams DB. Characteristics of smokers reached and recruited to an internet smoking cessation trial: a case of denominators. Nicotine Tob Res 2006 Dec;8 Suppl 1:S43-S48 [FREE Full text] [Medline]
- Graham AL, Fang Y, Moreno JL, Streiff SL, Villegas J, Muñoz RF, et al. Online advertising to reach and recruit Latino smokers to an internet cessation program: impact and costs. J Med Internet Res 2012;14(4):e116 [FREE Full text] [CrossRef] [Medline]
- Graham AL, Milner P, Saul JE, Pfaff L. Online advertising as a public health and recruitment tool: comparison of different media campaigns to increase demand for smoking cessation interventions. J Med Internet Res 2008;10(5):e50 [FREE Full text] [CrossRef] [Medline]
- Fenner Y, Garland SM, Moore EE, Jayasinghe Y, Fletcher A, Tabrizi SN, et al. Web-based recruiting for health research using a social networking site: an exploratory study. J Med Internet Res 2012;14(1):e20 [FREE Full text] [CrossRef] [Medline]
- Bauermeister JA, Zimmerman MA, Johns MM, Glowacki P, Stoddard S, Volz E. Innovative recruitment using online networks: lessons learned from an online study of alcohol and other drug use utilizing a web-based, Respondent-Driven Sampling (webRDS) strategy. J Stud Alcohol Drugs 2012 Sep;73(5):834-838. [Medline]
- Devineni T, Blanchard EB. A randomized controlled trial of an internet-based treatment for chronic headache. Behav Res Ther 2005 Mar;43(3):277-292. [CrossRef] [Medline]
- Andersson G, Bergström J, Holländare F, Carlbring P, Kaldo V, Ekselius L. Internet-based self-help for depression: randomised controlled trial. Br J Psychiatry 2005 Nov;187:456-461 [FREE Full text] [CrossRef] [Medline]
- Jones RB, O'Connor A, Brelsford J, Parsons N, Skirton H. Costs and difficulties of recruiting patients to provide e-health support: pilot study in one primary care trust. BMC Med Inform Decis Mak 2012;12:25 [FREE Full text] [CrossRef] [Medline]
- Leykin Y, Muñoz RF, Contreras O. Are consumers of Internet health information "cyberchondriacs"? Characteristics of 24,965 users of a depression screening site. Depress Anxiety 2012 Jan;29(1):71-77 [FREE Full text] [CrossRef] [Medline]
- Purdy CH. Using the Internet and social media to promote condom use in Turkey. Reprod Health Matters 2011 May;19(37):157-165. [CrossRef] [Medline]
- Gordon JS, Akers L, Severson HH, Danaher BG, Boles SM. Successful participant recruitment strategies for an online smokeless tobacco cessation program. Nicotine Tob Res 2006 Dec;8 Suppl 1:S35-S41. [Medline]
- Depression. NHS choices, your health, your choices URL: http://www.nhs.uk/conditions/depression/Pages/Introduction.aspx?WT.mc_id=61006 [accessed 2011-10-31] [WebCite Cache]
- Samaritans. 2011. Samaritans URL: http://www.samaritans.org/ [accessed 2011-10-31] [WebCite Cache]
- National Collaborating Centre for Mental Health.: National Institute for Clinical Excellence; 2004. Depression: The management of depression in primary and secondary care URL: http://www.nice.org.uk/nicemedia/pdf/cg023fullguideline.pdf [accessed 2012-11-20] [WebCite Cache]
- Kroenke K, Spitzer RL. The PHQ9: a new depression diagnostic and severity measure. Psychiatric Annals 2002;32(9 Online):0048-5713.
- National Health Service. Improving Access to Psychological Therapies URL: http://www.iapt.nhs.uk/ [accessed 2013-02-22] [WebCite Cache]
- R Development Core Team (2011). R: A language and environment for statistical computing URL: http://www.r-project.org/ [accessed 2012-11-20] [WebCite Cache]
- Jones RB, Goldsmith L, Hewson P, Kamel Boulos MN, Williams CJ. Do adverts increase the probability of finding online cognitive behavioural therapy for depression? Cross-sectional study. BMJ Open 2012;2(2):e000800 [FREE Full text] [CrossRef] [Medline]
- McCrone P, Dhanasiri S, Patel A, Knapp M, Lawton-Smith S. Paying the Price: The cost of mental health care in England to 2026. London: King's Fund; 2008. URL: http://www.kingsfund.org.uk/sites/files/kf/Paying-the-Price-the-cost-of-mental-health-care-England-2026-McCrone-Dhanasiri-Patel-Knapp-Lawton-Smith-Kings-Fund-May-2008_0.pdf [accessed 2013-02-22] [WebCite Cache]
- Singleton N, Bumpstead R, O'Brien M, Lee A, Meltzer H. Psychiatric morbidity among adults living in private households, 2000. Int Rev Psychiatry 2003;15(1-2):65-73. [CrossRef] [Medline]
|CBT: cognitive behavioral therapy|
|GP: general practitioner|
|IAPT: Improved Access to Psychological Therapies|
|IP: Internet protocol|
|LLTTF: Living Life To The Full|
|NHS: National Health Service|
|NICE: National Institute for Health and Clinical Excellence|
|OHFD: Online Help For Depression|
|PHQ9: Patient Health Questionnaire|
|RCT: randomized controlled trial|
Edited by G Eysenbach; submitted 25.09.12; peer-reviewed by J Powell, J Woodford, W van Ballegooijen; comments to author 09.11.12; revised version received 21.11.12; accepted 20.01.13; published 05.03.13
©Ray B Jones, Lesley Goldsmith, Paul Hewson, Christopher J Williams. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 05.03.2013.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.