This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Black and Hispanic men who have sex with men (MSM) are disproportionately affected by HIV in the United States. The Internet is a promising vehicle for delivery of HIV prevention interventions to these men, but retention of MSM of color in longitudinal Internet-based studies has been problematic. Text message follow-up may enhance retention in these studies.
To compare retention in a 12-month prospective Internet-based study of HIV-negative MSM randomized to receive bimonthly follow-up surveys either through an Internet browser online or through text messages.
Internet-using MSM were recruited through banner advertisements on social networking and Internet-dating sites. White, black, and Hispanic men who were ≥18, completed an online baseline survey, and returned an at-home HIV test kit, which tested HIV negative, were eligible. Men were randomized to receive follow-up surveys every 2 months on the Internet or by text message for 12 months (unblinded). We used time-to-event methods to compare the rate of loss-to-follow-up (defined as non-response to a follow-up survey after multiple systematically-delivered contact attempts) in the 2 follow-up groups, overall and by race/ethnicity. Results are reported as hazard ratios (HR) and 95% confidence intervals (CI) of the rate of loss-to-follow-up for men randomized to text message follow-up compared to online follow-up.
Of 1489 eligible and consenting men who started the online baseline survey, 895 (60%) completed the survey and were sent an at-home HIV test kit. Of these, 710 of the 895 (79%) returned the at-home HIV test kit, tested HIV-negative, and were followed prospectively. The study cohort comprised 66% white men (470/710), 15% (106/710) black men, and 19% (134/710) Hispanic men. At 12 months, 77% (282/366) of men randomized to online follow-up were retained in the study, compared to 70% (241/344) men randomized to text message follow-up (HR=1.30, 95% CI 0.97-1.73). The rate of loss-to-follow-up was non-significantly higher in the text message arm compared to the online arm for both white (HR=1.43, 95% CI 0.97-1.73) and Hispanic men (HR=1.71, 95% CI 0.91-3.23); however, loss-to-follow-up among black men was non-significantly lower among those who received text message follow-up compared to online follow-up (HR=0.78, 95% CI 0.41-1.50). In the online arm, black men were significantly more likely to be lost to follow-up compared to white men (HR=2.25, 95% CI 1.36-3.71), but this was not the case in the text message arm (HR=1.23, 95% CI 0.70-2.16).
We retained >70% of MSM enrolled in an online study for 12 months; thus, engaging men in online studies for a sufficient time to assess sustained outcomes is possible. Text message follow-up of an online cohort of MSM is feasible, and may result in higher retention among black MSM.
Over 60% of all new HIV diagnoses in the United States are among men who have sex with men (MSM) [
The recent increases in incidence among young MSM of color have led to a call for new approaches to HIV prevention [
Despite the benefits of Internet-based interventions, retention in online cohort studies of MSM has been problematic. In three online studies of MSM, 3-month retention was between 15%-54% [
Differences in retention in online studies by race/ethnicity might be partially explained by differences in Internet access. In 2011, approximately 66% of white Americans had broadband Internet access in the household, compared to 49% of black and 51% of Hispanic Americans [
Because young black and Hispanic Americans are high users of mobile technology, we sought to investigate whether the use of text messaging would increase retention in a 12-month online cohort study of HIV-negative white, black, and Hispanic MSM. Our primary aims were to compare the 12-month retention of MSM randomized to receive online follow-up surveys versus text message follow-up surveys and to compare 12-month retention by race/ethnicity. We hypothesized that providing follow-up surveys by text messaging would result in higher retention, especially among MSM of color. Additionally, we describe mobile access to online surveys and the frequency of changing mobile phone numbers.
MSM were recruited from August to December 2010 by banner advertisements placed on social networking and select Internet-dating websites, including Facebook, Myspace, Black Gay Chat, and Adam4Adam. Website selection was based on data from four focus groups of MSM conducted in 2010. We chose websites that men indicated that they visited frequently or sites on which they felt the advertisements were “legitimate” and “trustworthy”. Eligible participants were male, at least 18 years of age, white non-Hispanic, black non-Hispanic, or Hispanic, and reported sex with a man in the past 12 months. Additional eligibility criteria included owning a mobile phone capable of sending and receiving text messages, being willing to receive an at-home HIV test kit, and not moving outside the United States in the next 12 months. Because we were interested in determining the retention of an Internet-based sample of HIV-negative MSM, only those men who returned their HIV test kit and tested HIV-negative were followed prospectively for 12 months.
Men provided electronic informed consent prior to initiating any study procedures by checking a box on the survey screen. Consenting men were asked to register for the study by providing an email address and mobile phone number before completing the baseline survey. These were validated in two sequential steps. First, a unique participant-specific URL for the baseline survey was sent to the participant’s email address. Second, participants who successfully linked into the baseline survey through the URL in their email were asked to enter their mobile phone number, to which a 3-digit code was sent by text message. Participants entered the 3-digit code on a survey screen in order to proceed in the study.
Men with verified email and mobile phone information completed a 60-minute baseline survey that included questions on condom acquisition and use, demographics, sexual risk behaviors, sexual partner history, and HIV testing history. At the conclusion of the baseline survey, men who did not report being HIV-positive provided their mailing address for an at-home HIV test kit. Those who provided a valid mailing address were randomized 1:1 to receive either text message or online follow-up surveys every 2 months for a total of 12 months. Randomization was implemented through the online enrollment system; there were no blocks of randomization, so men were assigned to an arm through random number generation at the time each man was determined to be eligible. Participants were not blinded to the arm to which they were randomized. To facilitate completion of the follow-up surveys, we asked participants to choose a preferred day of the week and time of day to receive their follow-up surveys. Additionally, we requested that participants indicate a preferred alternate contact method, in the event that we were unable to contact them via email (for the online arm) or text message (for the text message arm).
Participants were compensated US $15 for completing the baseline survey, US $10 for each follow-up survey, and US $15 for the Month 12 survey. Men were also paid US $20 for returning their at-home test kit. Payments were delivered via PayPal or Amazon.com electronic gift card after completion of each survey. Participants randomized to the text message arm who did not have a text message plan from their mobile phone carrier were charged US $0.10 per text message response. The cost for text messages sent to participants was paid by the research team. Before providing informed consent, potential participants were informed about the potential to incur costs associated with sending text messages as part of the study.
Participants received notifications to take their follow-up surveys 8 weeks after their last completed survey. Participants randomized to receive online follow-up surveys received an email that contained a unique URL to link to the follow-up survey. Participants randomized to receive text message follow-up surveys received a text message that provided an opportunity for participants to initiate the survey immediately or delay the survey for 24 hours. The text message survey was a question-and-response format (ie, the subsequent survey question was only sent once the response to the previous question had been received). Similar to the online survey, the text message survey incorporated skip patterns based on participant responses so that only relevant questions were asked. The content of the follow-up surveys, which queried men on their 2-month sexual history and HIV testing history, was identical for both randomization arms. Regardless of randomization arm, all participants received an email notification for the final (Month 12) survey, which was administered online.
We used a systematic arm-dependent method to maximize retention. Men randomized to the online arm who had not completed the survey 3 days after the initial notification email were automatically sent a reminder email. Two subsequent automated reminder emails were then sent, each separated by 24 hours. Men randomized to the text message arm who did not initially complete the survey or did not request a delay of survey initiation received 3 additional text message reminders, each separated by 24 hours. Men in both randomization arms who did not complete a follow-up survey after the first group of reminders were contacted up to 3 additional times by study staff, using the preferred method of contact provided in the baseline survey. As a final step, study staff called the participant via mobile phone to remind him to complete his follow-up survey. Participants were withdrawn from the study if they did not complete the follow-up survey after 3 phone calls.
The primary outcome was loss to follow-up, defined as administrative withdrawal by study staff (for non-response, as described above), before the Month 12 survey, or request by a participant to be withdrawn from the study.
Using methods for time-to-event data [
Descriptive statistics were used to assess the distribution of participant characteristics by randomization arm, stratified by race/ethnicity. We used the Kaplan-Meier estimator to examine the rate of loss-to-follow-up by randomization arm and by race/ethnicity. We used Cox proportional hazards regression to estimate the hazard ratio (HR) and 95% confidence interval (CI) of time to loss-to-follow-up associated with randomization arm, overall, and stratified by race/ethnicity. We also estimated the HR and corresponding 95% CI of the rate of loss-to-follow-up within randomization arm for black and Hispanic participants relative to white participants.
We used scaled Schoenfeld residuals to evaluate the proportional hazards assumption of the Cox regression models [
All study procedures and analysis were reviewed and approved by the Institutional Review Board (IRB) of Emory University. This study used a randomized method for follow-up but did not meet the qualifications for ClinicalTrials.gov registration (ie, the study did not “prospectively assign human participants or groups of humans to one or more health-related interventions to evaluate the effects on health outcomes”.) This study did not utilize a health-related intervention or a health outcome.
There were 6174 MSM eligible to participate in the study (
Of 710 participants, 366 (52%) were randomized to online follow-up and 344 (49%) were randomized to text message follow-up. Two-thirds of participants were white and slightly over one-third were ≤24 years old (
Overall, 74% of men (523/710) were retained in the study at 12 months. Of the 187 men who were lost to follow-up, 18 (10%) requested to be withdrawn from the study and 169 (90%) were administratively withdrawn due to non-response. Withdrawal requests were more common among white men than black and Hispanic men: 13% (14/108) of white men, 8% (3/38) of black men, and 2% (1/41) of Hispanic men (
Nearly 10% (65/710) of men were lost to follow-up before the first follow-up survey (
Black men were less likely to be retained in the study compared to white or Hispanic men (
Approximately 20% (71/362) of white men, 22% (15/68) of black men, and 17% (16/93) of Hispanic men who completed the Month 12 online survey did so on a mobile browser. Additionally, in a 5-month period beginning in February 2011, 27 of 244 (8%) participants in the text message arm notified study staff that they had acquired new mobile phone numbers. This included 6% (14/229) of white participants, 11% (5/46) of black participants, and 12% (8/69) of Hispanic participants.
Enrollment of study participants in a 12-month prospective online study.
Baseline characteristics of participants in an online study, by race/ethnicity and randomization arm (N=710).a, b
|
|
|
White | Black | Hispanic | |||
Characteristic |
|
Total |
Online |
SMS |
Online |
SMS |
Online |
SMS |
|
||||||||
|
18-24 | 263 (37.0) | 71 (29.5) | 79 (34.5) | 28 (46.7) | 25 (54.4) | 28 (43.1) | 32 (46.4) |
|
25-34 | 262 (36.9) | 94 (39.0) | 81 (35.4) | 20 (33.3) | 16 (34.8) | 27 (41.5) | 24 (34.8) |
|
35-44 | 107 (15.1) | 43 (17.8) | 39 (17.0) | 7 (11.7) | 4 (8.7) | 6 (9.2) | 8 (11.6) |
|
45-54 | 78 (11.0) | 33 (13.7) | 30 (13.1) | 5 (8.3) | 1 (2.2) | 4 (6.2) | 5 (7.3) |
|
||||||||
|
≤ High school | 130 (18.3) | 41 (17.0) | 42 (18.3) | 14 (23.3) | 5 (10.9) | 10 (15.4) | 18 (26.1) |
|
> High school / GED | 580 (82.7) | 200 (83.0) | 187 (81.7) | 46 (76.7) | 41 (89.1) | 55 (84.6) | 51 (73.9) |
|
||||||||
|
West | 197 (27.7) | 56 (23.2) | 63 (27.5) | 9 (15.0) | 4 (8.7) | 25 (38.5) | 40 (58.0) |
|
Midwest | 96 (13.5) | 41 (17.0) | 32 (14.0) | 4 (6.7) | 6 (13.0) | 6 (9.2) | 7 (10.1) |
|
South | 278 (39.1) | 87 (36.0) | 90 (39.3) | 40 (66.7) | 26 (56.5) | 19 (29.2) | 16 (23.2) |
|
Northeast | 139 (19.6) | 57 (23.7) | 44 (19.2) | 7 (11.7) | 10 (21.7) | 15 (23.1) | 6 (8.7) |
|
||||||||
|
Urbane | 459 (66.6) | 150 (63.0) | 141 (63.2) | 47 (81.0) | 31 (73.8) | 46 (74.2) | 44 (66.7) |
|
Rural | 230 (33.4) | 88 (37.0) | 82 (36.8) | 11 (19.0) | 11 (26.2) | 16 (25.8) | 22 (33.3) |
|
||||||||
|
Homosexual | 603 (84.9) | 218 (90.5) | 201 (87.8) | 40 (66.7) | 31 (67.4) | 53 (81.5) | 60 (87.0) |
|
Bisexual | 88 (12.4) | 20 (8.3) | 23 (10.0) | 16 (26.7) | 9 (19.6) | 12 (18.5) | 8 (11.6) |
|
Other | 19 (2.7) | 3 (1.2) | 5 (2.2) | 4 (6.7) | 6 (13.0) | 0 (0.0) | 1 (1.5) |
Ever tested for HIV |
|
556 (78.5) | 195 (81.3) | 178 (78.1) | 45 (75.0) | 39 (84.8) | 52 (80.0) | 47 (68.1) |
HIV test in past 12m |
|
337 (47.6) | 118 (49.2) | 104 (45.6) | 26 (43.3) | 24 (52.2) | 35 (53.9) | 30 (43.5) |
|
||||||||
|
Men | 660 (93.0) | 229 (95.0) | 216 (94.3) | 55 (91.7) | 36 (78.3) | 60 (92.3) | 64 (92.8) |
|
Men and women | 50 (7.0) | 12 (5.0) | 13 (5.7) | 5 (8.3) | 10 (21.7) | 5 (7.7) | 5 (7.3) |
|
||||||||
|
1 | 102 (14.5) | 34 (14.1) | 33 (14.4) | 5 (8.6) | 7 (15.6) | 9 (14.1) | 14 (20.3) |
|
2-5 | 300 (42.5) | 99 (41.1) | 103 (45.0) | 27 (46.6) | 19 (42.2) | 25 (39.1) | 27 (39.1) |
|
>5 | 304 (43.1) | 108 (44.8) | 93 (40.6) | 26 (44.8) | 19 (42.2) | 30 (46.9) | 28 (40.6) |
UAI with MSP, past 12m | 556 (83.0) | 196 (86.0) | 181 (82.7) | 42 (73.7) | 33 (82.5) | 50 (80.7) | 54 (84.4) |
aOwing to missing data, numbers may not sum to column total. Denominators for proportions include those without missing data for that characteristic.
bAbbreviations—GED: general equivalency diploma; 12m: 12 months; (M)SP: (male) sex partner; UAI: unprotected anal intercourse.
cAs defined by the US Census Bureau.
dBased on zip code where participant requested that at-home HIV test kit was sent.
eUrban defined as residence in a zip code with population ≥1000 per square mile.
Retention of participants in an online study, by randomization arm (N=710).
Rate of loss-to-follow-up among men participating in a 12-month online study randomized to text message versus online follow-up, overall and stratified by race/ethnicity (N=710).
|
|
Proportion retained at 12 months |
Month 12 estimates | Month 10 estimates | ||
|
|
|
Hazard ratio | 95% CI | Hazard ratio | 95% CI |
|
||||||
|
Online | 282/366 (77.1) | Referent | — | Referent | — |
|
Text message | 241/344 (70.1) | 1.30 | 0.97-1.73 | 1.28 | 0.96-1.71 |
|
||||||
|
Online | 195/241 (80.9) | Referent | — | Referent | — |
|
Text message | 167/229 (72.9) | 1.43 | 0.97-2.09 | 1.41 | 0.96-2.06 |
|
||||||
|
Online | 37/60 (61.7) | Referent | — | Referent | — |
|
Text message | 31/46 (67.4) | 0.78 | 0.41-1.50 | 0.76 | 0.40-1.46 |
|
||||||
|
Online | 50/65 (76.9) | Referent | — | Referent | — |
|
Text message | 43/69 (62.3) | 1.71 | 0.91-3.23 | 1.67 | 0.89-3.16 |
aNumber retained out of total number defined by row.
Retention of participants in an online study, by race/ethnicity (N=710).
Rate of loss-to-follow-up among men participating in a 12-month online study randomized to text message versus online follow-up, by race/ethnicity and stratified by randomization arm (N=710).
|
|
Proportion retained at 12 months, n (%)a | Hazard ratio | 95% CI |
|
|
|
|
|
|
White | 362/470 (77.0) | Referent | — |
|
Black | 68/106 (64.0) | 1.70 | 1.17-2.45 |
|
Hispanic | 93/134 (69.4) | 1.37 | 0.96-1.97 |
|
|
|
|
|
|
White | 195/241 (80.9) | Referent | — |
|
Black | 37/60 (61.7) | 2.25 | 1.36-3.71 |
|
Hispanic | 50/65 (76.9) | 1.22 | 0.68-2.19 |
|
|
|
|
|
|
White | 167/229 (72.9) | Referent | — |
|
Black | 31/46 (67.4) | 1.23 | 0.70-2.16 |
|
Hispanic | 43/69 (62.3) | 1.47 | 0.94-2.35 |
aNumber retained out of total number defined by row
In this large Internet-based cohort of MSM, nearly three-quarters of men were retained in the study for 12 months. Black men randomized to text message follow-up were somewhat more likely to be retained than those randomized to online follow-up, but this was not the case for white or Hispanic men. We observed a similar rate of loss-to-follow-up among black and white men randomized to text message follow-up, but black men followed exclusively online were less likely to be retained than white men. Among all men, about 1 in 5 completed an online survey via mobile phone browser.
Our demonstrated retention of 74% at 12 months is encouraging to investigators wishing to establish large Internet-based cohorts of MSM. This proportion retained is over three times that observed by our group in a 3-month prospective of MSM in 2009 [
The high retention in our study can be attributed to a number of factors. First, we validated contact information from study participants, thereby excluding men who would have been lost to follow-up due to erroneous email addresses or mobile phone numbers. Second, based on results from our previous study which identified factors associated with retention in an online cohort of MSM [
Given the demonstrated need for new HIV prevention interventions among black MSM, it is promising that nearly two-thirds of black study participants were retained in this study. Although our 12-month retention of black men in this study is lower than the 3-month retention (78%) observed by Hightow-Weidmen et al at 3 months [
We were surprised that Hispanic men randomized to text message follow-up did not demonstrate a higher retention than those randomized to online follow-up, considering the high mobile phone usage by Hispanic Americans and relatively low household Internet access [
We noted that one-fifth of participants completed the Month 12 survey on their mobile phone. If the proportion of bimonthly online follow-up surveys completed via mobile Web is similar to that of the Month 12 survey, the higher retention in the online arm may be partially explained by mobile access to the survey. There are two important implications of this finding. First, research studies wishing to use Internet-based data collection may benefit from employing mobile-enabled Internet surveys. Second, studies should consider offering multiple methods of data collection. It is possible that men who completed surveys via mobile Web did so because they did not have household Internet access or because they preferred the convenience of a mobile phone. Either way, retention in studies may be enhanced by allowing participants to choose their preferred method of technology.
This study has a number of strengths. We enrolled a large geographically and ethnically diverse cohort of MSM recruited from multiple websites. We employed a novel text message data collection system that incorporated skip patterns and recognition of invalid data entries. We used an automated reminder survey system that delivered surveys at specific times requested by study participants and sent automatic reminders at set time intervals.
There are several limitations to this analysis. First, we defined our study sample based on an event (return of the at-home HIV test kit) that occurred after randomization. Therefore, we may have lost some of the benefit of randomization to balance arms on confounding factors. Although characteristics of our study population were relatively similar by arm within racial/ethnic group, we cannot assess the distribution of unmeasured confounders. Second, our final study population included men who completed the baseline survey, provided valid contact information, and returned an at-home HIV test kit. Therefore, our population likely represents an actively engaged sample of research participants for which retention may be optimized. Third, although we specifically targeted websites to enhance recruitment of minority MSM (eg, Black Gay Chat), enrollment of black and Hispanic men was below that of white men. This was disappointing, given that the goal of this study was to assess retention in an Internet-based cohort of minority men. However, this was not unanticipated, as we have previously characterized the under-enrollment of black and Hispanic MSM in online research [
In summary, we demonstrated an ability to retain >70% of MSM enrolled in an online study for 12 months. Our study suggests that follow-up via text message is feasible and may result in higher retention among black MSM. Based on our findings, it is possible to engage MSM at greatest risk for HIV infection in large prospective, Internet-based HIV prevention intervention studies using a time interval that is sufficient to assess sustained outcomes [
Coefficient estimate for randomization arm effect: online versus text message.
CONSORT-EHEALTH checklist V1.6.2 [
Centers for Disease Control and Prevention
Institutional Review Board
men who have sex with men
The authors would like to thank the men who participated in this study. We also thank Nicole Luisi for her data management support and Leonard Linde for his development and management of the text message system. The author(s) disclose receipt of the following financial support for the research and/or authorship of this article: National Center for Minority Health and Health Disparities (1RC1MD004370) and Emory Center for AIDS Research (P30 AI050409).
None declared.