Background: Rectal microbicides, if proven effective, may aid in reducing human immunodeficiency virus (HIV) incidence; however, demonstration of efficacy and effectiveness is contingent on accurate measurement of product adherence. Delays in self-report, in particular, may affect the accuracy of behavioral data.
Objective: The aim of this study was to capitalize on mobile phone use by young men who have sex with men (YMSM), and examine the use of an interactive voice response system (IVRS) by YMSM aged 18-30 years enrolled in a multisite, 12-week microbicide safety and acceptability trial.
Methods: YMSM (N=95) enrolled across 3 sites (Boston, Pittsburgh, and San Juan) were asked to report their use of an applicator applied placebo rectal gel product during receptive anal intercourse (RAI) using the IVRS. IVRS was available in Spanish and English. After the 12-week trial, we examined whether IVRS problems were associated with YMSM’s sociodemographic characteristics (eg, age, race and ethnicity, and education), sexual behavior, or recruitment site. We used a multinomial logistic regression to compare YMSM who experienced no IVRS problems (n=40) with those who reported one IVRS problem (n=25) or two or more IVRS problems (n=30).
Results: We recorded 1494 IVRS calls over 12 weeks. Over half of the participants (55/95; 58%) experienced challenges using the IVRS during the 12-week trial. YMSM reporting greater RAI occasions during the trial were more likely to experience one (odds ratio [OR]=1.08, 95% CI (1.02-1.14); P ≤.01) or more (OR=1.10, 95% CI (1.03-1.16); P ≤.001) IVRS challenges. Greater educational attainment was associated with multiple IVRS challenges (OR=7.08, 95% CI (1.6-31.6); P ≤.01). Participants in the Puerto Rico site were most likely to report multiple IVRS problems.
Conclusions: Although IVRS was a useful data collection technology in our trial, several challenges experienced by English and Spanish speaking YMSM diminish its overall acceptability. We discuss strategies to optimize future development of IVRS data quality protocols based on lessons learned.
The interactive voice reporting system (IVRS) has gained popularity given its potential to collect time-stamped, prospective behavioral data, and to reduce participants’ recall bias during data collection [- ] For example, participants can call into the system and report their data, rather than waiting until their next scheduled face-to-face visit with the research team [ , ]. IVRS has several additional methodological benefits to face-to-face approaches [ ]. These benefits include making data collection accessible at any time and from any place, easing individuals’ ability to participate in research if they experience scheduling or transportation-related barriers, and having greater cost-savings than face-to-face approaches. IVRS can also reduce literacy concerns by allowing participants to hear the questions and respond verbally and/or using a numeric keypad, allowing for assessments across multiple languages, and deploying complex, tailored skip-patterns based on participants’ answers [ ]. From a technological standpoint, IVRS also reduces compatibility issues as participants can use their phone of choice to report their behaviors and reduce data entry errors through automation [ ].
Researchers have used IVRS to monitor behaviors over short, intensive periods (eg, multiple assessments a day over a week) and to follow participants for longer study periods. Within HIV prevention and care, researchers have noted that IVRS is an acceptable method for participants to share sensitive information and might encourage greater privacy than face-to-face methods [- ]. In a study comparing Hispanic college students’ use of IVRS with a Self-Administered Questionnaire (SAQ) and a timeline follow-back (TLFB) over a 3-month interval, Schroeder and colleagues [ ] found that participants’ sexual and substance use behaviors were underreported in the TLFB and over-reported in the SAQ when compared with the daily IVRS reporting. In their analyses, however, Schroeder and colleagues noted that IVRS use varied based on participants’ sociodemographic characteristics (eg, age and sexual orientation), with younger and sexual minority participants trending toward a greater likelihood of placing IVRS calls than older and heterosexual counterparts [ ]. These findings suggest that participants’ characteristics might result in differential acceptability and use of IVRS.
Building on prior research examining the use of IVRS in HIV prevention studies, we examined the use of an IVRS to collect behavioral data from a sample of YMSM recruited to participate in a multisite rectal microbicide acceptability and adherence trial. Beyond overall use, however, we examined the prevalence of IVRS-related problems experienced by YMSM during the 12-week trial. We then examined whether YMSM’s sociodemographic characteristics (eg, age, sexual orientation, and educational attainment), sexual behaviors during the trial, and trial site (eg, Boston, Pittsburgh, and Puerto Rico) were associated with IVRS problems. We used our findings to discuss strategies to optimize future development of IVRS data quality protocols based on lessons learned in our trial.
Study data came from a larger project called Microbicide Safety and Acceptability in Young Men [, ]. The study received institutional review board (IRB) approval from all participating institutions, and all participants signed informed consent. After screening (stage 1A), YMSM participated in a run-in period in which they were asked to apply a rectal placebo gel using a rectal-specific applicator (stage 1B), followed by a safety trial in which participants applied tenofovir 1% gel using a vaginal applicator for rectal delivery of the gel (stage 2). The study took place in 3 sites: Pittsburgh, PA; Boston, MA; and San Juan, PR. Study candidates were recruited from clinics, bars, clubs, newspaper advertisements, and social networks. Recruitment materials indicated that the investigators were looking for YMSM (aged 18-30 years) for a study about their sexual health and their feelings about rectally inserting a placebo gel resembling a microbicide gel currently under development before receptive anal intercourse (RAI). Full protocol description (Clinicaltrials.gov NCT01283360) is presented in detail elsewhere [ ]. We focused our attention on stage 1B where the IVRS was used (December 2010 to October 2012).
Among participants who received medical clearance in stage 1A, we selected those fulfilling the more stringent eligibility criterion of having had condomless RAI within the prior 3 months to participate in stage 1B. This allowed us to focus on those with more recent potential risk and invite them to enroll in stage 1B. After undergoing an informed consent process, receiving risk reduction counseling and provision of condoms, and updating their medical history, participants received 20 rectal applicators filled with a placebo gel and instructions to insert the entire content of 1 applicator rectally within 90 min before each RAI episode. We used an applicator specifically designed for the delivery of a rectal microbicide  filled with hydroxyethylcellulose (HEC) gel. Hydroxyethylcellulose is also known as the “universal placebo” because of its use as placebo in most gel microbicide trials. Six weeks after their first stage 1B visit, participants returned for the mid-trial follow-up visit and were dispensed up to 20 additional applicators to ensure they had 20 on hand for the next 6 weeks. Six weeks after the mid-trial follow-up visit, participants returned for the final follow-up visit of stage 1B in which they completed a Web-based computer assisted self-interview (CASI) and semistructured interview that included questions on gel and applicator use. Participants received US $50 for each study visit (4 visits for stage 1A and 1B) and US $50 for a completed video teleconference interview. They also received US $1 per used applicator returned at visits 2 and 3.
Interactive Voice Response System (IVRS)
Over the 12 weeks of stage 1B, participants were instructed to call an IVRS after each instance of RAI and/or applicator insertion, or at least once a week if they did not have RAI. Participants could respond by voice or use their keypad. The IVRS system was available in English or Spanish. During visit 2, participants generated a 4-digit password to identify themselves within the IVRS and were trained on how to enter data into the IVRS, including completing a mock call into the IVRS. At each call, participants were asked to report the number of times they had used the gel, whether they had RAI or inserted anything other than the gel, and whether they experienced any problems with the IVRS. Participants who did not call into the system at least once a week were contacted by the IVRS and reminded to log their behaviors. Furthermore, to encourage use of the IVRS as a data collection tool, we incentivized participants with US $1 per call with a maximum of US $30 and an additional US $10 bonus a month if they called at least once a week. In total, participants could earn up to US $60 in stage 1B for reporting their product use via IVRS.
IVRS data was downloaded into an excel (Microsoft) spreadsheet at the end of the trial. We examined the frequency with which participants reported a problem when entering their data within the IVRS system, as well as the number of call entries that were incomplete (eg, hung up call) or sequential (eg, participant called into IVRS and had to call back again to revise [eg, mistyping] or finalize their entry). For this analysis, we coded an IVRS problem as a call where participants reported experiencing an IVRS problem or when an IVRS call was incomplete (eg, hung up call) or duplicative (eg, participant called again to complete their IVRS entry). Participants who indicated that they had an IVRS problem since their last completed call were given the option to leave a voicemail indicating what went wrong. A member of the study team transcribed these voicemails in English or Spanish. Spanish comments were subsequently translated into English. We include examples from these transcriptions in the Results section to illustrate participants’ problems when using the IVRS.
Sociodemographic data were collected via a Web-based CASI. Demographic information included age, race or ethnicity (white, African American, Latino, mixed or other), sexual orientation, and highest educational attainment (1=8th grade or lower, 2=some high school, 3=high school or general educational development [GED], 4=partial college, 5=college graduate, 6=some graduate school, and 7=graduate school degree). Participants also indicated whether they were currently in school (0=no; 1=yes) and employed (0=no; 1=yes). Participants were also asked to report the number of receptive anal intercourse occasions in the prior 3 months. Participants were identified by their site’s location (1=Pittsburgh, PA; 2=Boston, MA; and 3=San Juan, PR).
Data Analytic Strategy
After examining the descriptive statistics for our variables of interest, we used SPSS (version 23; IBM Corporation, New York) to test whether YMSM’s sociodemographic characteristics, sexual behavior, or site of recruitment were associated with the odds of experiencing IVRS problems during the 12-week trial. We used a multinomial logistic regression to compare the odds of experiencing a single or multiple IVRS problems during the 12-week trial. Due to limited observations in several categorical variables (eg, race or ethnicity and sexual orientation), we collapsed these indicators into dummy variables (eg, white: yes or no and gay-identified: yes or no) in our regression analyses.
Study participants (N=95) had a mean age of 23 years. The racial or ethnic composition of the sample was predominantly Latino (46/95, 48%) and white (33/95, 36%), followed by a fewer number of African American (9/95, 10%) and mixed or other race (7/95, 6%) participants. Most participants (86/95, 91%) identified as gay. Most of the sample (84/95, 88%) reported having at least some college education (see).
|Age (years), mean (SDa)||23.2 (3.2)|
|Race or ethnicity|
|White or European American||33 (34.7)|
|Black or African American||9 (9.5)|
|Latino or Hispanic||46 (48.4)|
|Mixed or other||7 (7.4)|
|Highest educational attainment|
|8th grade or lower||0 (0)|
|Some high school||1 (1.1)|
|High school graduate or GEDb||10 (10.5)|
|Some college||44 (46.3)|
|College graduate||27 (28.4)|
|Partial graduate school||2 (2.1)|
|Graduate school degrees||11 (11.6)|
|In school||47 (49.5)|
|Currently employed||60 (63.2)|
|San Juan||41 (43.2)|
|Receptive anal intercourse occasions (before 3 months), mean (SD)||17.0 (15.5)|
aSD: standard deviation.
bGED: general educational development.
The IVRS recorded 1494 calls from 95 participants over 12 weeks. We flagged 162 calls (162/1494, 10.8%) reflecting an IVRS problem. The most common problems resulted from mistyping a numeric answer (74/95 participants, 46%), experiencing challenges entering their answers into the system (27/95 participants, 17%), being disconnected midway through the call and having to call back (58/95 participants, 36%), or the system being inaudible (3/95 participants, 2%). We include examples of these IVRS problems using participants’ voicemail transcriptions in. Overall, 40 YMSM (42%, 40/95) did not have any trouble using the IVRS during the 12-week trial, 25 YMSM (26%, 25/95) had one problematic event across the 12-week trial, and the remaining 30 participants (32%, 30/95) reported two or more problems across the 12-week period.
|Type of problem||Frequency |
|Mistyping a number||74 (46)||“I mistakenly typed the wrong number; I pressed 1 when I should have pressed 2.” |
“I had to repeat questions several times because it wasn\'t getting the button presses.”
“Had to call 5-6 times because every time I tried entering my User ID, it would say ‘I’m sorry’.”
“(Problem with) entering my passcode.”
“The automated entry froze.”
“I mistakenly dialed that I didn’t use the gel during sex, but I did use it.”
|Being disconnected midway through the call||58 (36)||“I’ve called twice already and the call drops”. |
“Listen, I’ve tried to call 4 times and every time that I select an option, it indicates that I have nothing to say and hangs up on me. I’ve called 4 consecutive times. This time I pressed 1 to describe the situation.”
“Had to call back.”
“I was trying to record a message at the last time I called in and it stopped the message recording before I could say anything and it\'s happened more than once and I forget what I want to say the next time I call in because it doesn\'t let me re-record or anything.”
|Challenges reporting their answers into the system||27 (17)||“It didn’t let me log into the system. Four days later it called me to indicate that I hadn’t called in for that week”. |
“The only problem with the system is that it kinda takes a bit long to go through it and you have to wait for each question to finish makes it difficult to use after you call you know what buttons to press and basically you have to wait and the call takes longer.”
“I did not say why I did not use the gel it was probably because my roommate was in the room and I did not want to give a voice command but it was because I didn\'t have sex that I didn\'t use the gel.”
“When I gave my answer to several questions, it skipped and said: ‘I’m sorry, the answer is invalid’”.
“If there is any background noise while the phone system is waiting for me to press 1 or 2 it would say “I\'m sorry that was not a valid-I\'m sorry that was not a valid-I\'m sorry that was not a valid” and if it was completely quiet then it would stop doing that and I was able to press the buttons.”
|Inaudible system||3 (2)||“Your voice is not very clear at all, it\'s kinda blurry at some points.” |
“It was just cutting in and out.”
Using multinomial logistic regression (see), we examined whether IVRS problems were associated with participants’ sociodemographic characteristics, sexual activity, or study site. Compared with YMSM who did not experience any IVRS problems, YMSM who experienced one IVRS problem during the trial were more likely to have a greater number of RAI occasions during the 12-week trial. No other differences were observed between YMSM who experienced a single IVRS problem and those who had no IVRS problems.
YMSM who experienced two or more IVRS problems were more likely than YMSM without IVRS problems to report a greater number of RAI occasions during the 12-week trial, to report greater educational attainment, and to be currently in school. Participants in Puerto Rico were more likely to report two or more IVRS problems than peers in the Boston site. There were no differences between Boston and Pittsburgh. No other differences were observed between YMSM who experienced multiple IVRS problem and those with no IVRS problems.
|Characteristics||1 IVRS problem during 12-week period (N=25)||2+ IVRS problems during 12-week period (N=30)|
|OR||95% CI||Significance||OR||95% |
|RAI occasions (3 months)||1.08||1.02-1.14||.01||1.10||1.03-1.16||.001|
|−2LL=144.33; X218=42.5, P<.001||Nagelkerke Pseudo R2=44.0%|
Technological advances continue to increase researchers’ ability to capture behavioral data in experimental trials and observational studies. These data collection technologies (eg, IVRS, SMS text messages [short message service, SMS], and activity trackers) may be deployed to monitor participants’ behaviors and contexts outside of the clinic setting and have the potential to be synchronized with other systems (eg, electronic medical records and e-based applications that support behavior change) . In this study, we examined the use of an IVRS as a data collection tool in a rectal placebo gel acceptability and adherence trial with English and Spanish speaking YMSM. The IVRS recorded nearly 1500 calls over a 12-week trial period, highlighting its overall acceptability and feasibility among YMSM in HIV prevention studies.
Similar to other data collection methods, the use of IVRS as a data collection tool had its challenges. Ten percent of IVRS entries recorded during the trial were classified as having errors due to user (eg, user entering the wrong number to indicate their answer), system (eg, the IVRS not recognizing users’ voice responses when there is background noise), or connectivity (eg, bad cellphone signal causing a dropped call) issues. As such, researchers employing IVRS as a data collection tool should ensure that time and attention is placed on data quality assurances before protocol implementation and during data cleaning. Furthermore, most of the IVRS challenges observed in our study might decrease over time as new innovations emerge. Improvements in wireless infrastructure (eg, better signal strength across the globe), capabilities and programing of IVRS (eg, better voice-response accuracy and automated data cleaning clarification questions), and devices (eg, mobile phone features) may alleviate participants’ challenges when using an IVRS. For example, advances in IVRS programing could allow users to continue where they left off if they experience a problem (eg, call failure) and reduce frustrations stemming from having to restart their entry from the beginning. In situations when participants might feel uncomfortable verbalizing their answers (eg, someone walks into a room), designing opportunities for users to toggle their reporting through a multimodal response system (eg, switch from voice to text without interruption) may also improve IVRS data collection in real time.
Participants who reported greater RAI occasions during the trial were also more likely to experience IVRS challenges. Given that YMSM were instructed to call the IVRS every time they had RAI over the 12-week period, it is not surprising that those who contributed to a greater number of calls would have a greater likelihood of experiencing IVRS problems. After adjusting for participants’ sexual activity, however, we noted that the probability of experiencing one or more IVRS challenges differed across participants’ educational attainment and study site. YMSM with greater educational attainment reported more IVRS problems. These educational differences may be attributable to greater assertiveness to note problems with the system, and/or may be a proxy for socioeconomic differences regarding type of device (eg, mobile vs smart phone) owned. They may also have higher expectations about how efficient systems should work (they have more money, use high end devices, are more critical, and know what they can expect). In addition, participants in Puerto Rico were most likely to report having multiple IVRS challenges than peers in Boston. Although the IVRS had a Spanish version, it is possible that differences in participants’ speed and enunciation when speaking Spanish made it harder for the system to capture their data accurately. Compared with the mainland United States, optimal cell phone signal and connectivity in Puerto Rico is lower—particularly in the more rural areas of the island. Taken together, our results support Schroeder and colleagues’ findings  that participant sociodemographic characteristics may affect their use of the IVRS as a data collection tool.
Our study has several limitations that deserve a mention. First, we did not ascertain participants’ cellular plans or type of mobile phone during the study. Future research should consider how technological aspects (eg, type of phone and signal reliability) may affect data collection before IVRS implementation. In areas where signal strength or connectivity are a challenge, for instance, it may be better to rely on other methods (eg, an app notification system) that collects the information and stores it in the device in real time and subsequently transfers them to a central server once a connection is secured may be warranted. Third, our study focused on a sample of young men (ages 18-30 years) in 3 regions of the United States, limiting our ability to ascertain whether IVRS problems are similar or heightened in other age groups or contexts. Finally, our study took place from December 2010 to October 2012. The quality of IVRS is likely to have improved given the fast-pace of telecommunication innovations in society. Nevertheless, our study points to key data collection issues that must be considered as researchers plan and implement studies that rely on innovative data collection systems.
Overall, our study findings support the notion that IVRS is a feasible and acceptable method to collect time-stamped, prospective behavioral data from YMSM. As with other data collection methods, we encourage researchers to devote time and attention to the adequacy of IVRS for their populations of interest. Data quality assurances before protocol implementation, including considering how varying connectivity may create data collection challenges, are warranted. Nevertheless, even though some individuals might experience challenges, the interest and perseverance of YMSM to use this technology forecasts interesting possibilities that need to be explored in future research.
We greatly appreciate the hard work of the study staff at the sites and are indebted to the study participants for volunteering their time. This research was sponsored by the US National Institutes of Health (NIH), including National Institute of Child Health and Human Development (NICHD) and National Institute of Mental Health (NIMH), under R01 HD59533 (Carballo-Diéguez and McGowan, Co-PIs) and cosponsored by CONRAD. Additional support came from the NIHM to the HIV Center for Clinical and Behavioral Studies at NY State Psychiatric Institute and Columbia University (P30-MH43520; Principal Investigator: Robert Remien). The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH.
Conflicts of Interest
- Kennedy CM, Powell J, Payne TH, Ainsworth J, Boyd A, Buchan I. Active assistance technology for health-related behavior change: an interdisciplinary review. J Med Internet Res 2012 Jun 14;14(3):e80 [FREE Full text] [CrossRef] [Medline]
- Lee H, Friedman ME, Cukor P, Ahern D. Interactive voice response system (IVRS) in health care services. Nursing Outlook 2003 Dec;51(6):277-283. [CrossRef] [Medline]
- Ruikar V. Interactive voice/web response system in clinical research. Perspect Clin Res 2016;7(1):15-20 [FREE Full text] [CrossRef] [Medline]
- Fink JC, Doerfler RM, Yoffe MR, Diamantidis CJ, Blumenthal JB, Siddiqui T, et al. Patient-reported safety events in chronic kidney disease recorded with an interactive voice-inquiry dial-response system: monthly report analysis. J Med Internet Res 2016 May 26;18(5):e125 [FREE Full text] [CrossRef] [Medline]
- Steinberg DM, Levine EL, Lane I, Askew S, Foley PB, Puleo E, et al. Adherence to self-monitoring via interactive voice response technology in an eHealth intervention targeting weight gain prevention among Black women: randomized controlled trial. J Med Internet Res 2014 Apr 29;16(4):e114 [FREE Full text] [CrossRef] [Medline]
- Levin E, Levin A. Evaluation of spoken dialogue technology for real-time health data collection. J Med Internet Res 2006 Dec 11;8(4):e30 [FREE Full text] [CrossRef] [Medline]
- Brinkel J, Dako-Gyeke P, Krämer A, May J, Fobil JN. An investigation of users' attitudes, requirements and willingness to use mobile phone-based interactive voice response systems for seeking healthcare in Ghana: a qualitative study. Public Health 2017 Mar;144:125-133. [CrossRef] [Medline]
- Daftary A, Hirsch-Moverman Y, Kassie GM, Melaku Z, Gadisa T, Saito S, et al. A qualitative evaluation of the acceptability of an interactive voice response system to enhance adherence to isoniazid preventive therapy among people living with HIV in Ethiopia. AIDS Behav 2016 May 24:- Epub ahead of print. [CrossRef] [Medline]
- Schroder KE, Johnson CJ, Wiebe JS. Interactive voice response technology applied to sexual behavior self-reports: a comparison of three methods. AIDS Behav 2007 Mar;11(2):313-323. [CrossRef] [Medline]
- Schroder KE, Johnson CJ. Interactive voice response technology to measure HIV-related behavior. Curr HIV/AIDS Rep 2009 Oct 14;6(4):210-216. [CrossRef] [Medline]
- Tucker JA, Blum ER, Xie L, Roth DL, Simpson CA. Interactive voice response self-monitoring to assess risk behaviors in rural substance users living with HIV/AIDS. AIDS Behav 2012 Feb;16(2):432-440 [FREE Full text] [CrossRef] [Medline]
- Carballo-Diéguez A, Giguere R, Dolezal C, Bauermeister J, Leu C, Valladares J, et al. Adherence to rectal gel use among mainly ethnic minority young men who have sex with men during a 3-month placebo gel trial: implications for microbicide research. AIDS Behav 2014 Sep;18(9):1726-1733 [FREE Full text] [CrossRef] [Medline]
- McGowan I, Cranston RD, Mayer KH, Febo I, Duffill K, Siegel A, et al. Project gel a randomized rectal microbicide safety and acceptability study in young men and transgender women. PLoS One 2016;11(6):e0158310 [FREE Full text] [CrossRef] [Medline]
|GED: general educational developement|
|IRB: institutional review board|
|IVRS: interactive voice recognition system|
|RAI: receptive anal intercourse|
|SAQ: Self-Administered Questionnaire|
|TLFB: timeline follow-back|
|YMSM: young men who have sex with men|
Edited by G Eysenbach; submitted 10.03.17; peer-reviewed by J Galea, D Matthews, N Bragazzi; comments to author 19.04.17; revised version received 23.04.17; accepted 24.04.17; published 09.06.17Copyright
©Jose Bauermeister, Rebecca Giguere, Cheng-Shiun Leu, Irma Febo, Ross Cranston, Kenneth Mayer, Alex Carballo-Diéguez. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 09.06.2017.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.