This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
One of the advantages of Internet-based research is the ability to efficiently recruit large, diverse samples of international participants. Currently, there is a dearth of information on the behind-the-scenes process to setting up successful online recruitment tools.
The objective of the study was to examine the comparative impact of Spanish- and English-language keywords for a Google AdWords campaign to recruit pregnant women to an Internet intervention and to describe the characteristics of those who enrolled in the trial.
Spanish- and English-language Google AdWords campaigns were created to advertise and recruit pregnant women to a Web-based randomized controlled trial for the prevention of postpartum depression, the Mothers and Babies/
The Google ads received high exposure (12,983,196 impressions) and interest (176,295 clicks) from a global sample of Web users; 6745 pregnant women consented to participate and 2575 completed enrollment in the intervention study. Keywords that were descriptive of pregnancy and distress or pregnancy and health resulted in higher consent and enrollment rates (ie, high-performing ads). In both languages, broad keywords (eg, pregnancy) had the highest exposure, more consented participants, and greatest cost per consent (up to US $25.77 per consent). The online ads recruited a predominantly Spanish-speaking sample from Latin America of
The Internet is an effective method for reaching an international sample of pregnant women interested in online interventions to manage changes in their mood during the perinatal period. To increase efficiency, Internet advertisements need to be monitored and tailored to reflect the target population’s conceptualization of health issues being studied.
ClinicalTrials.gov NCT00816725; http://clinicaltrials.gov/show/NCT00816725 (Archived by WebCite at http://www.webcitation.org/6LumonjZP).
The amount of Internet-based research in recent years has increased substantially [
Most Internet intervention studies have focused on depression and anxiety, but an emphasis on other health issues is gaining interest as further evidence of their effectiveness is demonstrated [
Women and women-specific health issues are well suited for technology-based interventions with perinatal women garnering much of the attention by researchers (eg, [
Although the Internet offers added benefits and opportunities, there are significant concerns and barriers with recruitment to Internet-based research investigations [
What remains unclear in the literature is how researchers should frame their Internet advertisements so that they are time-efficient and cost-effective (ie, targeted to a specific population who are most likely to enroll in the study). Graham and colleagues [
The current study adds to the growing body of literature describing online recruitment methods for Internet intervention studies. The primary goal was to examine the impact of Google AdWords campaigns to recruit Spanish- and English-speaking pregnant women to an Internet RCT to prevent PPD. Both the Spanish and English ad campaigns used identical keywords and were minimally modified throughout the recruitment period. Given that few Internet intervention studies have focused exclusively on pregnant women, we describe the demographic characteristics of participants who enrolled in the trial.
Google AdWords sponsored link campaigns were created in Spanish and English to advertise the Mothers and Babies/
The Google AdWords account management website allows the owner of the ads to manipulate the bidding cost and content of each ad. A Google AdWords grant awarded to our team allowed for a maximum bid of US $1.00 per click. Ad owners can view the number of times the ad is shown (impressions) and the number of Web users who click on the ad (clicks); owners can also use HyperText Markup Language (HTML) code to track the impact of the ads on obtaining the desired action by the target audience (conversion). In this study, the HTML code to detect a conversion was integrated into the study website’s consent page (ie, 1 conversion = 1 consent to participate). For the purposes of this paper, we will refer to conversions as “consents” given that it is the desired outcome of online recruitment. Additional information on Google AdWords procedures and metrics can be found elsewhere [
Google ads in Spanish and English for a prevention of postpartum depression trial.
Web users who searched the Internet between February 3, 2009 and June 15, 2012 with keyword queries associated with the ads were able to click on them and link to the Mothers and Babies/
Screenshot of the study landing page.
Upon entering the study website, participants indicated their country of residence and preferred language and completed items to determine their eligibility to participate (age, gender, pregnancy status, how they planned to use the website materials). Eligible participants who entered a valid email address were directed to the baseline assessment.
The initial page of the survey was the informed consent that participants were required to complete before proceeding through the study website. Consenting participants were asked to complete questionnaire items on demographic characteristics (eg, country of birth, ethnicity, race, education), Internet use (eg, previous use for health information), pregnancy history (eg, weeks pregnant, pregnancy history), and depression (eg, current symptoms).
Depression status was determined by the Center for Epidemiologic Studies-Depression Scale (CES-D) [
Data on the performance of search engine ads were extracted from the Google AdWords management website [
Web users interested in the Spanish- and English-language Google ads, as defined by clicking on the ad (ie, clicks), were located in 183 countries and territories; 6745 pregnant women consented to participate with 2517 (37.32%) failing to enter any data in the baseline assessment. Data collected in the eligibility screener indicated that those who consented but did not provide baseline information were older (
The Google ads were active for approximately 40 months during which 12,983,196 impressions and 176,295 clicks were made in response to Web users’ search entries (see
The keywords with the greatest exposure (ie, highest clicks and impressions) and which generated the highest traffic in both languages were pregnancy/
Participant online recruitment for the Mothers and Babies/
Ad or user behavior | Language, n | Total, n | |
|
Spanish | English |
|
Impressions (ads presented) | 4,898,063 | 8,085,133 | 12,983,196 |
Clicks (clicks on ads) | 116,531 | 59,764 | 176,295 |
Entered site (user proceeded beyond initial page) | 28,074 | 16,157 | 44,231 |
Screened (user answered ≥1 eligibility items) | 11,620 | 5349 | 16,969 |
Eligible (user met eligibility criteria) | 8728 | 3738 | 12,466 |
Consented (user agreed to consent) | 4773 | 1972 | 6745 |
Enrolled (user consented and completed baseline assessment) | 2012 | 563 | 2575 |
Performance of Google AdWords campaigns from February 3, 2009 to June 15, 2012.
Ad or user behavior | Ad performance, n | |||||
|
Sad | Depressed | Postpartum depression | |||
|
Spanish | English | Spanish | English | Spanish | English |
Consent | 4855 | 1782 | 136 | 179 | 1 | 2 |
Clicks | 114,902 | 53,667 | 1574 | 5987 | 55 | 110 |
Impressions | 4,794,470 | 7,184,384 | 92,280 | 856,634 | 11,313 | 44,115 |
Spanish- and English-language ads with the highest-performing keyword(s).
Ad character headline | Keyword(s) | Consents | Cost per consent |
Clicks | Impressions | |
|
|
|
|
|
|
|
|
|
|
4718 | $15.98 | 113,525 | 4,819,662 |
|
|
|
156 | $8.55 | 2130 | 54,987 |
|
|
|
109 | $2.89 | 739 | 11,716 |
|
|
|
|
|
|
|
|
Sadness during pregnancy | Pregnancy/pregnant | 1823 | $25.77 | 57,197 | 7,836,146 |
|
Depression and pregnancy | Pregnancy depression | 82 | $11.99 | 1301 | 97,488 |
|
Pregnancy and sadness | Sad pregnant | 17 | $12.52 | 290 | 18,095 |
|
Pregnant and depressed? | Depressed pregnant | 16 | $9.12 | 193 | 16,457 |
The final sample of enrolled participants (N=2575) consisted of pregnant women with a mean age of 28.16 years (SD 5.47) (see
There were no differences in pregnancy status or depression history between the Spanish- and English-speaking participants. Participants were mostly in the second trimester of their pregnancy (mean 16.56 weeks, SD 9.59) and most (69.34%) did not meet
Group comparisons revealed that the Spanish-speaking women were more likely than English-speaking women to self-identify their ethnic background as Latino or Hispanic (91.45 vs 9.85%,
Baseline characteristics of enrolled participants.
Demographic items | Spanish |
English |
Total |
|
|
Age, mean (SD) | 28.18 (5.68) | 28.07 (4.60) | 28.16 (5.47) | .69 | |
Latino/Hispanic (n=2343,a), n (%) | 1818 (91.45) | 39 (9.85) | 1857 (77.89) | <.001 | |
|
|
|
|
<.001 | |
|
|
819 (44.10) | 5 (1.03) | 824 (35.17) |
|
|
European/Caucasian descent | 608 (32.74) | 105 (21.60) | 713 (30.43) |
|
|
Other | 305 (16.42) | 73 (15.02) | 378 (16.13) |
|
|
Asian descent | 11 (0.59) | 204 (41.97) | 215 (9.18) |
|
|
African descent | 12 (0.65) | 96 (19.75) | 108 (4.61) |
|
|
American Indian/Alaska Native | 102 (5.49) | 3 (0.62) | 105 (4.48) |
|
Married/live with partner (n=2572), n (%) | 1224 (60.92) | 499 (88.63) | 1723 (67.99) | <.001 | |
|
|
|
|
<.001 | |
|
12 years or less | 353 (17.86) | 68 (12.32) | 421 (16.65) |
|
|
University level/degree | 1450 (73.38) | 329 (59.60) | 1779 (70.37) |
|
|
Advanced degree | 173 (8.75) | 155 (28.08) | 328 (12.97) |
|
Employed (n=2561), n (%) | 1217 (60.76) | 317 (56.81) | 1534 (59.90) | .09 | |
Weeks pregnant (n=2564), mean (SD) | 16.71 (9.62) | 16.01 (9.47) | 16.56 (9.59) | .13 | |
|
|
|
|
.61 | |
|
None | 1371 (69.31) | 357 (69.45) | 1728 (69.34) |
|
|
Current MDE | 372 (18.81) | 103 (20.04) | 475 (19.06) |
|
|
Past MDE | 235 (11.88) | 54 (10.50) | 289 (11.60) |
|
CES-D score (n=2475), mean (SD) | 27.86 (13.89) | 26.07 (13.22) | 27.51 (13.77) | .01 | |
|
|
|
|
|
|
|
During perinatal period | 873 (43.39) | 225 (39.96) | 1098 (42.64) | .15 |
|
When not pregnant (n=2591) | 643 (32.05) | 107 (19.35) | 750 (29.31) | <.001 |
aValid percent reflects participants who completed the item.
b
Country of residence of enrolled participants.
Country | %a | |
|
|
|
|
Chile | 16.30 |
|
Mexico | 15.24 |
|
Spain | 12.00 |
|
Argentina | 11.35 |
|
Colombia | 11.20 |
|
Venezuela | 11.00 |
|
Peru | 5.23 |
|
Bolivia | 3.43 |
|
Ecuador | 2.79 |
|
Dominican Republic | 2.54 |
|
Uruguay | 1.34 |
|
Paraguay | 1.14 |
|
29 Countries/territories with less than 1% each | 6.44 |
|
|
|
|
India | 36.30 |
|
South Africa | 12.81 |
|
Pakistan | 7.47 |
|
United Kingdom | 7.12 |
|
Iran | 2.67 |
|
Nigeria | 2.31 |
|
United States | 2.13 |
|
Ghana | 1.96 |
|
Kenya | 1.78 |
|
United Arab Emirates | 1.60 |
|
Saudi Arabia, Maldives, Uganda, Ireland (each) | 1.07 |
|
54 Countries/territories with less than 1% each | 19.57 |
aBased on self-reported country of residence data.
This study examined the impact of Spanish- and English-language Google AdWords to recruit an online sample of participants to an Internet RCT for the prevention of PPD. This method of recruitment was effective at exposing a large number of female Web users to the opportunities offered by the study website, such as learning skills to manage changes in mood during the transition to motherhood. During the 3 years that the ads were active, 176,295 Web users clicked on the advertised link and 12,466 pregnant women met eligibility criteria; 54% agreed to participate and 2575 (20.66%) completed sufficient baseline information to enroll in the study. These data suggest that pregnant women from around the world are interested in learning skills to manage their mood during and after pregnancy, are willing to use Web-based resources, and that the Internet is a viable means to reach them. Although a large number of women showed interest in joining the study, a large proportion exited the site immediately after clicking the ad or failed to continue once they were informed of participation details. A possible explanation for the former is the mismatch between what Web users queried and what they eventually found on the study’s landing page. That is, the information listed on the site did not correspond with what they were looking for or hoping to find when they searched, clicked, and initially visited the study website. The burden of participation may have been a factor to dissuade eligible participants from continuing their engagement in the study once they learned the details of participation from the consent form. Clearly, we need to make such Internet research sites more interesting and less burdensome so that more of those who are eligible not only consent to participate but also complete all facets of the study.
Our examination of the highest-performing keywords for each of the Google AdWords campaigns revealed 3 primary findings. First, keywords that were more descriptive of the website content (eg, pregnant and depressed) resulted in a higher rate of consent to participate. This pattern of user behavior was evident in both the Spanish and English ads that referenced pregnancy and emotions, and suggests that the biggest gains occur when advertisements closely relate to the product being promoted. This approach is especially important online where there is heavy competition for the user’s attention and sustaining interest in one site over another is challenging. The specificity of pairing a reference to pregnancy and emotions in the keywords resulted in a higher consent rate among Spanish versus English users. This keyword type was also the most cost-effective in both languages relative to other high-performing keyword types (eg, pregnancy without referencing emotions). Cost-effectiveness by itself is not the most important measure of usefulness. The most cost-effective keywords yielded the least number of participants. To recruit the most participants in a reasonable period of time, we must be ready to pay for keywords that yield higher cost per consent. Given that most researchers are working within budgetary constraints, being able to maximize the cost of each consented individual is of high priority and importance. Furthermore, for the English ads there were more variations of this keyword type and, therefore, more related activity. Given that both the Spanish and English ads were identical in their content and associated keywords, this finding may suggest that Spanish-speaking women are less likely to search for information related to emotional health during pregnancy than English-speaking women or that the associated keywords do not reflect how Spanish-speaking women conceptualize perinatal distress. In fact, keywords that did not reference emotions or feelings but described pregnancy characteristics (eg, months pregnant) were very high activity ads and yielded a much larger number of impressions, clicks, consents, and enrolled participants. This suggests that women may be initially attracted by information about pregnancy that does not include information on emotional health during pregnancy. Thus, ads that are more generic may result in better recruitment outcomes for more women, of whom a subset may be subsequently intrigued by the focus on such issues as mood.
Previous reports have suggested that Latinos and Spanish speakers manifest psychological distress through physical complaints that can stem from stigma, cultural barriers, and differences in how psychological issues are conceptualized and experienced [
A secondary aim of this study was to examine the characteristics of individuals who consented to participate in a Web-based trial. The ads attracted a mostly Spanish-speaking sample of women who resided in Latin American countries. This was an unexpected outcome of the recruitment efforts given the lower Internet penetration rate in this region of the world relative to Oceania/Australia, Europe, and North America [
Spanish- and English-speaking participants were relatively similar on demographic characteristics with a few exceptions. The breakdown of the origin of our participants is important within a global mental health perspective, and especially critical when considering maternal mental health needs worldwide. To date, few resources exist in these regions of the world to address the day-to-day needs of pregnant women [
Participants in this study were well educated, with a greater proportion of English-language speakers possessing advanced degrees, whereas Spanish-language speakers had mostly attended university level or fewer years of formal education. It is clear that women with varied levels of education are using the Internet to obtain pregnancy-related information. Spanish- and English-speaking participants differed in their marital status with Spanish speakers more likely to indicate that they were single or without a current partner, thus potentially raising their risk for the onset of postpartum mood disorders [
Our findings were limited in several ways. First, the Google AdWords grant awarded to our team capped our bids at US $1.00. Second, this study solely reports on the use of minimally modified Google AdWords campaigns. We do not know how the ads would have performed if we had managed the campaigns based on their performance by adjusting the keywords, daily cost, or regional exposure. Third, given how the ads were set up and linked to the study website, we are unable to connect keyword types to participant characteristics. In order to understand how to target the ads to participants in different regions of the world who speak different languages and who conceptualize emotional distress differently, it would be beneficial to explore this further by recording the keyword entered by the user in their online search and examining their characteristics and behaviors on the study website. Finally, our data are only generalizable to Spanish and English speakers who use the Web to obtain information on the perinatal period.
This study contributes to the growing understanding of online recruitment for intervention trials. We provide evidence that pregnant women in many regions of the world are already seeking this type of health information and choosing to engage in Internet interventions by virtue of their participation in this study. Nonconsumable Internet interventions or those that can be reused with minimal added cost have the potential to reduce health disparities globally because they can be used an unlimited number of times from any region of the world without significant increases in cost [
Center for Epidemiological Studies-Depression
HyperText Markup Language
major depressive episode
postpartum depression
randomized controlled trial
The original RCT was funded by the National Institute of Mental Health (F32MH077371). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health or the National Institutes of Health. Additional funding was provided by a Robert Wood Johnson Health Disparities Seed Grant (Nancy Adler, PI) to Dr Barrera, funding from the University of California Committee on Latino Research which funded the UCSF/SFGH Latino Mental Health Research Program (Muñoz, PI), and funds granted to the UCSF Internet World Health Research Center (Muñoz, Director) from the Department of Psychiatry at SFGH while under the direction of Robert Okin, MD. A very special thanks to Google, Inc for awarding the team (Muñoz, PI) an AdWords grant, which provided the ability to recruit worldwide using Google-sponsored links. Online data were collected by using the Illume survey system developed by DatStat, Inc. We thank Drs Lynn Abeita and Eunice Kim for feedback on earlier drafts of this paper. Lastly, we are thankful to all the women who participated in this research.
None declared.