This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Raising awareness of online cognitive behavioral therapy (CBT) could benefit many people with depression, but we do not know how purchasing online advertising compares to placing free links from relevant local websites in increasing uptake.
To pilot a cluster randomized controlled trial (RCT) comparing purchase of Google AdWords with placing free website links in raising awareness of online CBT resources for depression in order to better understand research design issues.
We compared two online interventions with a control without intervention. The pilot RCT had 4 arms, each with 4 British postcode areas: (A) geographically targeted AdWords, (B) adverts placed on local websites by contacting website owners and requesting links be added, (C) both interventions, (D) control. Participants were directed to our research project website linking to two freely available online CBT resource sites (Moodgym and Living Life To The Full (LLTTF)) and two other depression support sites. We used data from (1) AdWords, (2) Google Analytics for our project website and for LLTTF, and (3) research project website. We compared two outcomes: (1) numbers with depression accessing the research project website, and then chose an onward link to one of the two CBT websites, and (2) numbers registering with LLTTF. We documented costs, and explored intervention and assessment methods to make general recommendations to inform researchers aiming to use similar methodologies in future studies.
Trying to place local website links appeared much less cost effective than AdWords and although may prove useful for service delivery, was not worth pursuing in the context of the current study design. Our AdWords intervention was effective in recruiting people to the project website but our location targeting “leaked” and was not as geographically specific as claimed. The impact on online CBT was also diluted by offering participants other choices of destinations. Measuring the impact on LLTTF use was difficult as the total number using LLTTF was less than 5% of all users and record linkage across websites was impossible. Confounding activity may have resulted in some increase in registrations in the control arm.
Practitioners should consider online advertising to increase uptake of online therapy but need to check its additional value. A cluster RCT using location targeted adverts is feasible and this research design provides the best evidence of cost-effectiveness. Although our British pilot study is limited to online CBT for depression, a cluster RCT with similar design would be appropriate for other online treatments and countries and our recommendations may apply. They include ways of dealing with possible contamination (buffer zones and AdWords techniques), confounding factors (large number of clusters), advertising dose (in proportion to total number of users), record linkage (landing within target website), and length of study (4-6 months).
clinicaltrials.gov (Registration No. NCT01469689); http://clinicaltrials.gov/ct2/show/NCT01469689 (Archived by WebCite at http://www.webcitation.org/6EtTthDOp)
Less than 60% of people with diagnosable depression or anxiety seek formal help from practitioners; this represents a significant treatment gap [
There is increasing evidence that online interventions can be effective in changing health behaviors or improving health [
However, many patients do not benefit through lack of awareness. We found a variation in registration to LLTTF of 15-fold between the highest (Kirkwall, Scotland) and lowest (Wigan, England) postcode areas [
Raising awareness of online CBT could benefit many people with depression by facilitating early rapid access. Ways of addressing this include online advertising through search engines such as Google AdWords (AdWords) [
AdWords have been used by others to recruit participants to studies, for example, for depression screening [
Studies that compare recruitment methods before and after interventions have no control group that could be described and compared with adequately. The only rigorous way of comparing methods of raising awareness of online therapies is by geolocated cluster randomised controlled trials (RCT, [
We planned to compare purchasing AdWords with the second strategy of using weblinks and carried out a pilot RCT of the two different recruitment interventions to check the methods and outcome measures for a definitive trial later. In particular, our study objectives were to explore: (1) whether or not the two recruitment interventions seemed to work at all, and so be worthy of further study, (2) the ability to target online Google adverts or weblinks from other local websites without contamination, (3) the ability to link data sources to effectively measure the impact of the recruitment interventions, and (4) to learn more about the likely size and impact on target CBT site use and, given the other methodological issues such as confounding factors, to know what sample size and dose of advertising will be needed for future substantive studies.
The study was approved by the NHS South West 2 Research Ethics Committee (Reference 11/H0203/8; February 2011) and registered on clinicaltrials.gov (Registration No. NCT01469689).
This was a pilot cluster RCT of recruitment interventions for online CBT for depression. We compared two online recruitment interventions with a control without intervention. The pilot RCT had 4 arms: (A) geographically targeted paid AdWords, (B) free adverts/weblinks placed on local websites by contacting those website owners (weblinks), (C) both interventions combined, (D) control. Participants were directed to a project recruitment website linking to two freely available online CBT sites (Moodgym and LLTTF) and two other sites (Samaritans [
Schematic of the design showing websites and data sources (in gray).
All 121 postcode areas in England, Wales, and Scotland were divided into quartiles by rate of registration on LLTTF (based on 36,753 registrations of people between June 2008 and June 2009 who scored 8 or more on the Hospital Anxiety and Depression Scale for either anxiety or depression) and into quartiles by population size (based on the 2001 census). We randomly selected 4 cells from this array, and chose 4 nearly consecutive postcode areas for each cell (to try to achieve similar populations), avoiding adjacent geographical areas. Each set of 4 postcode areas was randomly allocated to the 4 arms of the trial (three interventions and control). Randomization was not blind.
Each arm of the study had a total population ranging from 1.6 to 2 million people clustered in 4 postcode areas. In total the study included 7 million people in 16 postcode areas across England, Wales, and Scotland. The estimated point prevalence for major depression among 16- to 65-year olds in the UK was 2.1%, rising to 9.8% when the less specific and broader category of “mixed depression & anxiety” was included [
Those paying for AdWords campaigns set up one or more adverts, and enter keywords to help determine when the advert is shown. AdWords displays adverts as a sponsored link, either at the top of the list of search results or in the right hand search results panel, depending on the phrase entered, the price offered per advert, bids from competing adverts, and (if requested) by estimated location of the user.
Within the 8 areas in arms A and C, we ran AdWords from April 17th to November 30th, 2011. In the 8 areas in B and C, we aimed to place adverts (weblinks) from local organization websites such as local universities, general practitioner (GP) practices, and local authorities, by contacting these organizations via email and/or phone starting on April 17th. Arm D was a control arm with no recruitment intervention.
We used a single advert (
Google Advert.
We originally asked AdWords to display the advert for the keyword of depression. AdWords suggested other similar keyword combinations and we accepted all suggestions. AdWords gave information on the number of times they presented adverts against Google searches, by day, keyword, and location. AdWords decided when to present the advert based on the price we offered, the price of competing adverts, and other factors such as the search terms used. Users searching on terms such as depression and depression help were presented with our advert, depending on our budget and competing adverts.
Google searches were used to identify organizations in 8 postcode areas (arms B and C) with websites to place adverts (weblinks to our project website) on local free access websites. In our search we included websites such as local GP surgeries, local media websites including newspaper, TV, radio, further and higher education institutions, and community-based or local charity websites. In total, 180 emails were sent with 3 weblinks posted free of charge: two in Leeds (university medical practice and a carers’ organization), and one in Kirkwall (local online community newspaper).
Those who clicked on Google adverts or on weblinks were directed to our project research website OHFD. This gave information about the study and advised that completion of the online questions implied consent for it to be used in the study (see
We used various sources of data to model patient flow (
We documented costs for using AdWords and other weblinks and estimated costs per person with depression referred to the online CBT via AdWords, based on time spent, cost £10/hour (based on the hourly rate Plymouth University pays temporary administrative staff) and Google’s charges. We included costs needed for routine delivery of these methods and excluded research costs such as time spent in the comparison of methods or in setting up the research project website.
We defined 2 main outcomes and compared them between the 4 arms: (1) the numbers accessing the project research website (OHFD) that completed a PHQ9 depression rating score and had a score of more than 5, indicating at least mild depression, and then chose an onward link to one of the two CBT websites, and (2) the numbers registering with LLTTF who gave their postcode, comparing intervention with control and all areas. In the revised version of LLTTF (issued in January 2011) used in this study, new arrivals at LLTTF do not have to immediately register, instead, registration can be delayed. Registration, by giving more personal details and agreeing to email reminders, signifies a commitment to use the site seriously. In the previous version of LLTTF used to select the sample, most visitors to the site registered as they could not access most content until after registration.
We carried out this study at a time when the Improved Access to Psychological Therapies (IAPT) project [
Examining the number of people clicking through from online adverts to OHFD to online CBT (
It proved impossible to directly track individuals from OHFD to LLTTF using IP address and time data, therefore the impact of the interventions on LLTTF had to be assessed using Analytics and LLTTF website data. Analytics reported 1474 visits landing on LLTTF from OHFD, agreeing approximately with the 1581 referred by OHFD, although the 7% attrition is unexplained (
We know from Analytics that the bounce rate (ie, those who exited from the first page) for visits from OHFD was 47% (no different to the average bounce rate from referred visits of 49%). This suggests that those arriving from OHFD were not more or less likely to continue and to subsequently register. We can see from the difference between arrivals on the site (14,396 from Analytics) and registrations (1143 from LLTTF log data) from the study arms that 10% (1067/10569) of all visitors registered on the current version of LLTTF.
Participant flow diagram showing overall recruitment and different data sources (April 17th - November 30th 2011). Shaded boxes show numbers for the two outcomes.
Number of people on OHFD with depression (PHQ>5) choosing online CBT and LLTTF, registering on LLTTF, annual rate per 100,000 estimated depressed registering on LLTTF before and during interventions, by postcode area, and trial arm.
|
|
Outcome 1 |
Outcome 2 |
||||||
|
|
Intervention period |
Before intervention |
Intervention period |
|||||
|
Estimate of people with |
People with |
Annual rate per 100,000 |
People with depression who chose LLTTF | Rega
|
Annual rate per 100,000 depressed |
Rega
|
Annual rate per 100,000 depressed |
|
|
|
|
|
|
|
|
|
|
|
|
Liverpool |
42,173 | 155 | 588 |
111 | 38 | 360 |
129 | 489 |
|
Redhill |
24,721 | 49 | 317 |
34 | 59 | 955 |
94 | 608 |
|
Lancaster |
16,299 | 42 | 412 |
27 | 44 | 1080 |
73 | 717 |
|
Harrogate |
6668 | 7 | 168 |
6 | 19 | 1140 |
40 | 960 |
|
Total | 89,860 | 253 | 450 |
178 | 160 | 712 |
336 | 598 |
|
|
|
|
|
|
|
|
|
|
|
Leeds |
36,867 | 24 | 104 |
16 | 69 | 749 |
171 | 742 |
|
Southend |
24,660 | 12 | 78 |
8 | 55 | 892 |
130 | 843 |
|
Slough |
16,882 | 10 | 95 |
6 | 22 | 521 |
59 | 559 |
|
Kirkwall |
2505 | 1 | 64 |
0 | 17 | 2715 |
30 | 1916 |
|
Total | 80,914 | 47 | 93 |
30 | 163 | 806 |
390 | 771 |
|
|
|
|
|
|
|
|
|
|
|
London |
39,167 | 104 | 425 |
77 | 86 | 878 |
170 | 694 |
|
Kingston |
24,505 | 72 | 470 |
50 | 36 | 588 |
75 | 490 |
|
Darlington |
17,074 | 21 | 197 |
15 | 37 | 867 |
78 | 731 |
|
Shetland |
1099 | 1 | 146 |
1 | 5 | 1819 |
18 | 2620 |
|
Total | 81,846 | 198 | 387 |
143 | 164 | 802 |
341 | 667 |
|
|
|
|
|
|
|
|
|
|
|
Nottingham |
54,012 | 10 | 30 |
8 | 102 | 755 |
272 | 806 |
|
Oldham |
22,190 | 3 | 22 |
1 | 42 | 757 |
69 | 498 |
|
Dudley |
19,882 | 4 | 32 |
4 | 30 | 604 |
56 | 451 |
|
Hebrides |
1325 | 0 | 0 |
0 | 4 | 1207 |
15 | 1811 |
|
Total | 97,409 | 17 | 28 |
13 | 178 | 731 |
412 | 677 |
Study total | 35,0028 | 515 | 235 |
364 | 665 | 760 |
1479 | 676 |
|
Other areas | 25,04345 | 1791 | 114 |
1130 | 5613 | 897 |
11894 | 760 |
|
E, W, and S | 28,54373 | 2306 | 129 |
1581b | 6278 | 880 |
13,373 | 750 |
aregistrations
b1581 includes 87 who gave no postcode on OHFD, but all other indications (eg, IP address) show that they were England (E), Wales (W), or Scotland (S).
As a validity check on location, we carried out an alternative analysis on those registering on LLTTF using the Analytics location data instead of user stated postcodes. The two approaches showed agreement.
According to Analytics, the biggest increase in “landings” on LLTTF was seen in one of the control areas, Nottingham.
Ignoring the stated locations of those clicking through to LLTTF, we see that 1581 clicked through to LLTTF in the study period (
Number of visits to LLTTF according to Google Analytics. Top graph shows Liverpool, middle Nottingham, and the bottom graph shows ALL UK visits between January and November 2011.
Just under 8000 (7980/8231, 97%,
All 3 weblinks were for locations in arm B. Arm C therefore was effectively another AdWords arm. The total cost of the AdWords campaign in payments to Google was £1841. From April 17th to October 19th, we set a daily budget of £7.50 (4.4 pence per 1000 population) and had one campaign in which all 8 areas were included. As we described elsewhere [
We estimated that the time spent simply adjusting the AdWords campaign, as opposed to time spent on Analytics trying to match with other data, was 25 hours. At £10/hour, the total cost of the AdWords campaign was £2091 (£1841+£250). If we assume that both arms A and C were AdWords, this represents £4.64 per person for the 451 people who chose online CBT and had a PHQ9 >5 in arms A and C (the target group,
Participant flow diagram showing randomization to the 4 arms of the study.
Payments to Google for AdWords campaign.
Area | Maximum daily |
Average daily spend | Clicks | Cost | Cost per click | ||||
|
|||||||||
|
All | £7.50 | £7.07 | 6291 | £1,301.04 | £0.21 | |||
|
|||||||||
|
Liverpool | £3.84 | £2.37 | 365 | £142.45 | £0.39 | |||
|
London SW | £3.56 | £2.48 | 886 | £148.83 | £0.17 | |||
|
Redhill | £2.24 | £1.19 | 182 | £71.65 | £0.39 | |||
|
Kingston | £2.22 | £1.43 | 174 | £85.99 | £0.49 | |||
|
Darlington | £1.56 | £0.76 | 66 | £45.72 | £0.69 | |||
|
Lancaster | £1.48 | £0.68 | 64 | £40.61 | £0.63 | |||
|
Harrogate | £0.60 | £0.08 | 21 | £4.60 | £0.22 | |||
|
Shetland | £0.60 | £0.00 | 1 | £0.29 | £0.29 | |||
|
Total | £16.10 | £9.00 | 1759 | £540.14 | £0.31 |
In total, we were able to recruit 12 panel members and all arms, but not all postcode areas, of the study were represented by either an ex-user of LLTTF or a member of an IAPT team. No other local interventions were identified via the user panel but the panel was able to help in examining the search environment across Britain [
McCrone estimated the number of people with depression in England as 1.24 million (2.3%) with total cost of services at £1.7 billion while lost employment increased this total to £7.5 billion [
Various ways of raising awareness of websites are available including online advertising and weblinks. We did not know if online adverts such as AdWords would be more or less effective than weblinks. These methods may simply attract the same people that would in any case have found online CBT. As a pilot study, we were looking at the feasibility of being able to answer these questions and what study design would enable us to do so.
We have found that using AdWords is possibly going to be cost effective, whereas trying to get weblinks is not worth pursuing further in this locality based research study design. We only managed to put 3 weblinks in place. Scaling up the AdWords campaign would incur further advertising but no further labour costs (so the unit price gradually decreases), whereas scaling up the weblinks campaign would be proportional to the labour costs. Establishing weblinks therefore appears much less cost effective than AdWords and is not worth pursuing as a sole intervention in this study design.
However, the ability to establish weblinks may be significantly easier for local organizations trying to offer local services. In our study, we chose areas remote from the research team, and there were no prior local relationships on which to build. Being locally situated and delivering local services to local people might significantly affect the ease of establishing local websites. Others have used weblinks to successfully recruit as part of their recruitment package. For example, 91% (174/191) of participants in a trial of online treatment for chronic headache learned of the study from weblinks (both mutual and paid for) on websites, registration with major search engines, and notices posted to headache-related news groups, but the authors did not compare these methods and the study was global [
AdWords were effective in recruiting people to the project website but our pilot study identified a number of issues for our, and similar, cluster RCTs. The main problems were leakage so that its effect was greatly diluted, problems in linking data across websites so that the impact of advertising was lost in the large numbers using LLTTF, and possible confounding factors. The dose of advertising (particularly given the “record linkage” problem) and length of study also need to be considered. The changes that would be needed to the design of a definitive trial are discussed below. These findings will be relevant to others seeking to improve the uptake of online interventions or designing online cluster RCTs.
There was leakage at 3 points in the process. First, we have shown elsewhere [
To decide whether a definitive trial will be feasible, we modelled the effect of reduced leakage. By taking 50% of the number lost on the project website before making a destination choice, by routing all participants to LLTTF, and by reducing losses from location targeting from 76% to 20%, we would reduce the leakage from 94% to 47%.
We were not able to track individuals from OHFD to LLTTF and therefore could only use the overall data to try to estimate outcome 2, the number of people registering with LLTTF. Although outcome 1 showed significant differences between intervention and controls, to be able to measure a difference, the number referred from AdWords needed to be sufficiently larger compared to the total number that registered on LLTTF. The number of people with depression referred from OHFD to LLTTF was 4.7% (498/10569) of all users of LLTTF in study arms. If, as above, leakage was reduced to 47% the number referred by OHFD would still only be 10% of all those landing on LLTTF. To be certain that an impact of AdWords can be seen, the dose of advertising also needs to be increased. Although we doubled the daily budget to 9.3 pence per 1000 target population for the last two months of this pilot study, the number of clicks was slightly less than in the first period, probably due to breaking the advertising into 8 separate campaigns and AdWords not having had time “to settle”. The effectiveness of the AdWords advertising budget depends on competing adverts, and is likely to be less cost effective as the daily budget is increased. As shown in
A major contributing factor to the lack of impact on outcome 2 was the increase in uptake of LLTTF in one of the control areas (Nottingham), possibly as a result of a presentation given to practitioners by one of the authors (CW) to a national conference that overlapped with a surge of use. Although it was not a blinded region selection, day-to-day management of the study was conducted by RJ. We recognized in designing the pilot study that random interventions or effects from other influences, such as local campaigns, could impact on intervention or control areas. For that reason, we had attempted to monitor all areas via the user panel. This pilot study suggested that, to avoid an overdue influence of one postcode area (cluster), each arm needs many more than 4 clusters. As we had designed the pilot study with 2 types of interventions and 4 arms, we only used 16 out of 120 postcode areas for our study.
A more robust approach would be to use all postcode areas in Britain, excluding London postcodes, ordered by population size only, randomized in pairs to two arms, excluding any adjacent postcode areas as buffer zones around study areas. Using this method produces a two-arm study with 32 postcode areas (see
We paid for over 8 months of advertising but the recruitment numbers stabilized after a few months and we did not change the advert for 6 of the 8 months. On the other hand, it takes 2-3 months for AdWords to reach peak efficiency. With reduced leakage, a bigger sample, and an increased daily budget on advertising, the cost of a definitive study could probably be reduced by examining changes over 6 months.
Assessing location is subject to error. Previously [
Studies that aim to change Internet use will be limited by frequent changes to websites and other technical advances. In our case, even though CW was the author of LLTTF, changes underway to the LLTTF website could not be postponed for this study. Our sample was selected based on data extracted from LLTTF between the years 2008 and 2009 when most users of the site registered. In January 2011, after our study had been designed and ethical approval had been sought, LLTTF was reconfigured such that registration was optional and could be done at a later time. As a result, the number of people that registered greatly reduced. This meant that registration figures were no longer directly comparable with those collected earlier. Although we were able to compare the relative differences between regions in 2009 with 2011, by creating an index of use based on the lowest use region, we restricted direct comparison (ie, the “before” period) to January- April 2011.
Graham [
In this pilot study we selected our sample trying to match for the populations of postcode areas and the baseline registration rates on LLTTF. Although it would be important not to have a grossly imbalanced sample, the need for a greater number of postcode areas (as described above) while ensuring geographical dispersion to avoid contamination would seem to be more important in the design.
Although various online methods exist to raise awareness, the problem of being able to select sample areas allowing for the competing demands of contamination and confounding factors suggest that having a factorial design with more than one recruitment intervention would be difficult. On the other hand, for practitioners seeking to increase recruitment using a mix of all possible methods, more than one recruitment intervention would seem to be sensible.
Our pilot study confirms other research that many people search online for help with mental health issues [
In discussing this research with NHS policy makers, the response to advertising, perhaps in the light of previous criticisms of their expenditure, was that NHS websites have been optimized and therefore appear high in search results, so there is no need for online advertising. This may be true but our exploration of this [
Although the results of the weblinks in this pilot study was not as expected and will be excluded from this research design (unless a better method of placing local weblinks can be found), it would be wrong to conclude that weblinks are not relevant for practitioners and policy makers. The problem may be that the type of site containing relevant links tends to be either national (inappropriate for a location targeted clusters RCT) or very local (researchers likely do not have prior knowledge about existence of these sites). Practitioners and policy makers may therefore have to rely on weaker evidence from before/after studies of cost and impact to decide on how much effort they put into using weblinks.
Part of the reason why LLTTF had greater use in Scotland compared to other parts of Britain is that professionals in those areas were more aware of the existence of these sites and recommended use of these sites to patients. The role of professionals in recommending online CBT may also have explained the confounding results seen in one area in the control group. So it may be that continuing professional development and raising awareness among professionals about resources for depression may be as, or more effective, than direct-to-patient online interventions. However only one third of patients were in contact with health services and other studies have shown that trying to reach patients to tell them about online resources may be labour intensive, time consuming, and very expensive [
A cluster geo-located RCT to test the cost-effectiveness of online advertising seems feasible in Britain. Geolocated adverts are offered by Google in other countries based on a radius around a point, so the general design of a cluster RCT to test the cost effectiveness of online advertising in raising awareness of an online therapy, as piloted in this study, would seem to hold true for other countries and for other online therapies or websites. However, this pilot study has demonstrated 4 general messages concerning contamination, confounding, dose of advertising, and length of study that will be useful for other researchers.
Provided it is possible to have a sufficiently large buffer zone between intervention and control regions, it should be possible to deal with potential contamination. The definition of sufficiently large is vague and needs to be piloted in each country, but in Britain, we think a 2-arm trial with 16 postcode areas in each should be possible without too much contamination. The problems of including London in Britain were described in more detail elsewhere [
Ideally, such a trial would have a large number of clusters in each arm such that any one region is kept to approximately 5% of the total population in the arm. That would mean that any confounding activity such as local media campaigns would not influence the arm greatly. However, dealing with confounding factors conflicts, to some degree, with the relatively small and densely populated country like Britain. In our proposed best design for Britain, trying to keep contamination to a minimum at the postcode area level, our design of 2-arms of 16 postcode areas each, still included a city representing 17% of that arm. This was slightly risky for introducing confounding factors but was the best we could do. In a bigger country such as the United States, it should be possible to have a stronger design. In a smaller, densely populated country such as the Netherlands, this study design may be impossible.
We recommend that in such a design, adverts are linked to a special “landing page” within the target site. This allows those following the link to track their use of the target site. We had used an intermediate project website in this pilot because we were offering 2 online CBT sites and 2 other sites. This gave us considerable problems in trying to track participants from the advert to the project website to the target website, and in trying to match up the different data sets available from AdWords, Analytics, and website logs. Having a landing page within the target website should remove most problems but it would still help in the design if the dose of advertising, and so the expected “footfall” from advertising was substantial compared to the number arriving from other sources. In this pilot, with a limited advertising budget, although we had significant numbers arriving at our project website, the problem of leakage meant that the number of people finding the target website compared to the numbers finding it from a normal search, national weblinks, or direct entry were small. Any study with this type of design should pay greater attention to the total footfall on the target website and try to calculate the dose of advertising needed to have a demonstrable impact.
The cost of advertising in such a trial will be partly determined by how long the campaign is run. We ran our pilot for 7.5 months. Although it did take several weeks for our campaign to stabilize and for AdWords to get the best return and cost per click, a campaign of 4-6 months should be sufficient for this type of study where the total number clicking is in the range of 300-500 people. The best design (as described above) is to have a larger dose of advertising over a shorter period rather than a small dose over a longer period.
We cannot be sure how the findings of this study would translate to other countries and there is no guarantee that location targeting of online advertising will continue to be available in this form. Increasing use of mobile phones may change the way location targeted adverts work. AdWords is of course not the only way of local advertising online and a definitive trial might consider use of advertising solutions from Microsoft, Facebook, LinkedIn, and others. Facebook, for example, offers location targeting and would also be worth exploring in this way.
Our study was limited by the difficulties of trying to match different data sources. Different sources from Google (AdWords and Analytics) do not exactly match due to different ways of collecting the data. Other issues, because of the anonymity of the data, include whether visitors are unique individuals. For example, although Analytics may claim to report unique visitors, we cannot verify that claim and it would be impossible for Analytics to differentiate between two individuals using the same computer (IP address) and one person with two emails using the same computer. LLTTF only collects emails of those who register. It is likely, therefore, that all sources overestimate the number of unique individuals. However, although numbers from different sources do not match exactly, the overall picture seems consistent and reasonably robust.
This pilot study has shown that a definitive cluster trial of AdWords is worthwhile, and that this type of design could be used to assess other online recruitment interventions.
Screenshots from Online Help For Depression.
Two arms for definitive cluster randomized trial, showing the 16 postcode areas in each arm, cluster populations, and total population for each arm.
cognitive behavioral therapy
general practitioner
Improved Access to Psychological Therapies
Internet protocol
Kingston
Living Life To The Full
National Health Service
National Institute for Health and Clinical Excellence
Online Help For Depression
Patient Health Questionnaire
randomized controlled trial
We would like to thank Ian Mayer of Mirata Ltd for help in extracting anonymised data from LLTTF, Robert Stillwell for developing the project website. RJ developed the idea for the study, was the principal investigator and grant holder of the study. RJ was responsible for day-to-day management of the project, carried out most of the analysis, wrote and edited the paper. LG contributed to the research proposal, was a co-grant holder, undertook day-to-day management, did some of the analysis, helped write and edit the paper. PH reviewed the whole study, in particular the statistical methods in the paper, suggested an alternative method to model the data, carried out that modelling, and edited the paper. CW contributed to the research proposal, was co-grant holder, and edited the paper. The project was funded by a grant from the Bupa Foundation Philip Poole-Wilson Seedcorn Fund. Ethics committee approval reference: 11-H0203-8.
Chris Williams is the designer and author of the LLTTF site, and is Director of Five Areas Ltd which licences the LLTTF website. LLTTF is run under licence by the charity Jubilee Trust.