This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Longitudinal follow-up of older persons living with HIV is essential for the ascertainment of aging-related clinical and behavioral outcomes, and self-administered questionnaires are necessary for collecting behavioral information in research involving persons living with HIV. Web-based self-reported data collection results in higher data quality than paper-and-pencil questionnaires in a wide range of populations. The option of remote web-based surveys may also increase retention in long-term research studies. However, the acceptability and feasibility of web-based data collection in clinical research involving older persons living with HIV have never been studied.
This study aims to assess the acceptability and feasibility of a web-based survey to collect information on sexual, substance use, and physical activity behaviors; compare the data quality of the web-based survey with that of a paper-and-pencil questionnaire; and summarize web-based survey metrics.
This pilot study took place within the AIDS Clinical Trials Group A5322 study, a longitudinal cohort of men and women living with HIV (aged ≥40 years), followed at 32 clinical sites in the United States and Puerto Rico. A total of 4 sites participated in this study. A web-based survey was created using self-administered questionnaires typically completed in A5322 via paper and pencil. Pilot study participants completed these questionnaires via web-based survey at one research visit in lieu of paper-and-pencil administration. Two questions were added to assess feasibility, defined as participants’ perception of the ease of web-based survey completion (very hard, hard, easy, very easy), and their preferred format (computer or tablet, paper and pencil, no preference) for completing the questions in the future (acceptability). Feasibility and acceptability were summarized overall and by demographic and clinical characteristics; the proportion of evaluable data by web-based survey versus previously administered paper-and-pencil questionnaires (data quality) was compared for each question.
Acceptability and feasibility were high overall: 50.0% (79/158) preferred computer or tablet, 38.0% (60/158) reported
Web-based survey administration was acceptable and feasible in this cohort of older adults living with HIV, and data quality was high. Web-based surveys can be a useful tool for valid data collection and can potentially improve retention in long-term follow-up studies.
Maintaining sustained research participation is a critical challenge for longitudinal epidemiologic studies [
Web-based data collection methods that allow the flexibility of remote survey completion may help improve retention in long-term studies [
The AIDS Clinical Trials Group (ACTG) A5322 study is examining a wide range of clinical and behavioral end points in individuals aging with HIV. Data collection methods include interviews, physical examinations, and chart abstraction as well as self-administered questionnaires that are completed by hand (paper-and-pencil format). Once completed, these questionnaires are handed to research clinic staff in sealed envelopes and mailed to a data management center where responses are keyed. This process can result in incomplete, missed, or lost questionnaires. In addition, although these forms were designed for ease of completion, they do contain free-text responses as well as specific instructions, including those for skip patterns. Participants may have difficulty following the instructions, and the resulting data can be of poor quality because of issues such as skipping questions inappropriately and entering out-of-range free-text responses. Opportunities to clean these data are inherently limited, as data managers cannot query site staff or participants regarding their responses, and as a result, some information in these forms cannot be used.
We piloted a web-based survey that adapted 3 paper-and-pencil questionnaires administered in ACTG A5322. Our overall objective is to determine whether behavioral data collected via paper-and-pencil format could be successfully collected using web-based surveys. This was designed as the initial step of a longer-term goal of incorporating web-based surveys into the study’s regular schedule of evaluations, including potential expansion into remote data collection. Our specific aims are to (1) assess the acceptability and feasibility of the web-based survey and identify demographic and health characteristics associated with these measures; (2) compare the data quality of the web-based survey with that of the paper-and-pencil questionnaire; and (3) summarize web-based survey metrics, including frequency of and reasons for survey noncompletion and frequency of
The A5322 study is an ongoing, long-term observational study following older men and women living with HIV for characterization and evaluation of age-related outcomes. Participants had previously been followed in ACTG A5001, another long-term observational study of participants who had received either their initial HIV antiretroviral treatment medication (treatment naive) or a salvage therapy through an ACTG randomized clinical trial. When the A5001 follow-up ended, participants who had been treatment naive at the time of enrollment in their initial ACTG clinical trial and were aged ≥40 years were eligible to enroll in the A5322 study. Altogether, 1035 participants were enrolled between November 2013 and July 2014 at 32 clinical research sites across the United States, including Puerto Rico. Participants were previously evaluated semiannually (now annually) for immunologic, virologic, and clinical parameters and annually for behavioral parameters. All participants provided written informed consent before enrollment into A5322, and the study was approved by the local institutional review board at each site.
A total of 4 sites were chosen for this pilot study based on the number of participants they had enrolled into A5322, the proportion of participants who spoke English as their first language, and the availability of a laptop or desktop with wired internet and access to a private space in which to complete the survey. All participants at the 4 sites whose primary language was English were eligible to participate in the pilot study. The web-based survey was administered during a single visit, which took place at the 4 sites, in place of the paper-and-pencil questionnaires typically administered.
The survey was developed using Illume, a commercial software tool designed by DatStat (DatStat, Incorporated). The survey was developed only in English and consisted of 3 questionnaires on recent sexual behaviors, current and past substance use, and physical activity (the latter using the International Physical Activity Questionnaire Short Form [
Frequency of and reasons for noncompletion of the survey were collected on a tracking form completed by the clinic staff. Acceptability was assessed with a three-category variable from the question, “In the future, I would prefer to complete the survey by
The data quality of the web-based survey and paper-and-pencil responses was assessed by comparing the proportion of evaluable responses to each question in the web-based survey with that obtained from previously administered paper-and-pencil questionnaires. Evaluable data were defined as any valid response, excluding
Demographic and functional fitness characteristics were assessed at the visit closest to the survey administration unless otherwise indicated. The variables included age, race/ethnicity, sex, education (assessed during A5001 follow-up), history of comorbidities (diabetes, kidney disease, liver disease, cardiovascular disease, stroke, hepatitis C–positive serology, and cancer [within 5 years]), frailty, disability; and neurocognitive function and impairment. Frailty was assessed using the Fried Frailty assessment, which includes 4-m walk speed; grip strength; and self-reported unintentional weight loss, exhaustion, and low activity [
To compare the frequency of
We compared the distribution of acceptability by demographic and health characteristics using chi-square tests for categorical variables and the Kruskal-Wallis test for continuous variables. Multinomial logistic regression models were fit to evaluate the association of these variables with acceptability as a three-category variable,
We compared the proportion of evaluable responses to each question by survey format using chi-square tests. The first comparison was between pilot study participants’ web-based responses and their responses to the most recently completed paper-and-pencil questionnaire. A second comparison was between pilot study participants’ web-based responses and responses of all A5322 participants on their most recently completed paper-and-pencil questionnaire. Chi-square tests were used for these comparisons instead of a matched approach because the number of responses to many of the questions differed by mode of administration. Therefore, it was not possible to use a matched approach that would take into account within-person correlations.
Finally, we compared the proportion of evaluable data for all questions within the web-based survey by age and neurocognitive impairment.
SAS 9.4 was used for all analyses (SAS Institute).
A total of 180 participants at the 4 sites were eligible to participate in the pilot study. Of these 180 participants, 159 (88.3%) completed the web-based survey; for the 21 eligible participants who did not complete the survey, the following reasons were provided: participant declined (8/180, 4.4%), clinic error (7/180, 3.9%), technical difficulties (3/180, 1.7%), time constraints (2/180, 1.1%), and cognitive impairment (1/180, 0.6%).
Of the 35 questions in the web-based survey that included a
Demographic and functional fitness by acceptability.
Characteristic | Prefer computer/tablet (n=79) | Prefer paper-and-pencil questionnaire (n=19) | No preference (n=60) | Total (N=158a) | ||
Age (years), median (Q1-Q3) | 53 (48 to 60) | 59 (52 to 66) | 55 (50 to 61.5) | 54 (49 to 61) | .04b | |
Sex, malec, n (%) | 62 (79) | 14 (74) | 53 (88) | 129 (81.7) | .21d | |
|
.75d | |||||
|
White, non-Hispanic | 43 (54) | 13 (68) | 39 (65) | 95 (60.1) |
|
|
Black, non-Hispanic | 30 (38) | 6 (32) | 19 (32) | 55 (34.8) |
|
|
Hispanic (regardless of race) | 3 (4) | 0 (0) | 2 (3) | 5 (3.2) |
|
|
Asian, Pacific Islander | 2 (3) | 0 (0) | 0 (0) | 2 (1.3) |
|
|
More than one race | 1 (1) | 0 (0) | 0 (0) | 1 (0.6) |
|
NPZ-3 score: median (Q1-Q3)e | 0.55 (−0.20 to 1.20) | 0.50 (−0.20 to 1.20) | 0.75 (0.10 to 1.25) | 0.60 (−0.10 to 0.20) | .34b | |
Neurocognitive impairmente,f, n (%) | 9 (11) | 2 (11) | 4 (7) | 15 (9.5) | .59d | |
Greater than high school education levelf, n (%) | 48 (61) | 12 (63) | 43 (72) | 103 (65.2) | .55d | |
Frail, n (%) | 8 (10) | 1 (5) | 5 (8) | 14 (8.9) | .79d | |
≥1 IADLg limitationf, n (%) | 19 (24) | 2 (11) | 4 (7) | 25 (15.8) | .02d | |
History of any comorbidity, n (%) | 68 (86) | 16 (84) | 52 (87) | 136 (86.1) | .96d |
aOne participant (Black non-Hispanic male, high school education, no IADL limitations) skipped the acceptability question.
bKruskal-Wallis test.
cSex at birth. Information on gender not available.
dChi-square test.
eBlack, White, and Hispanic participants only.
fThe following variables had missing observations: neurocognitive impairment (N=3, all chose computer/tablet), education (N=3, all chose computer/tablet), and IADL limitations (N=1, chose
gIADL: Instrumental Activities of Daily Living.
Overall, 50.0% (79/158) of participants indicated that they would in the future prefer to answer the questionnaires via computer or tablet, 12.0% (19/158) said they would prefer the
In an adjusted multinomial logistic regression model including age and IADL, with
Multinomial logistic regression model: factors associated with acceptability (paper and pencil as the reference group).
Variable and outcome | ORa (95% CI) | ||
|
|||
|
Computer/tablet | 0.91 (0.85-0.98) | .01 |
|
No preference | 0.95 (0.89-1.02) | .14 |
|
|||
|
Computer/tablet | 2.72 (0.55-13.5) | .22 |
|
No preference | 0.62 (0.1-3.74) | .60 |
aOR: odds ratio.
bIADL: Instrumental Activities of Daily Living.
Multinomial logistic regression model: factors associated with acceptability (no preference as the reference group).
Variable and outcome | ORa (95% CI) | ||
|
|||
|
Computer/tablet | 0.96 (0.92-1.01) | .11 |
|
Paper-and-pencil questionnaire | 1.05 (0.98-1.13) | .14 |
|
|||
|
Computer/tablet | 4.42 (1.4-13.9) | .01 |
|
Paper-and-pencil questionnaire | 1.62 (0.27-9.87) | .60 |
aOR: odds ratio.
bIADL: Instrumental Activities of Daily Living.
Most participants (147/158, 93.0%) reported that the web-based survey was easy or very easy to complete. Individuals who found the survey hard or very hard had a lower median NPZ-3 score than did those who found it easy or very easy; age was not associated with feasibility (
Age and NPZ-3 score by feasibility.
Characteristics | Hard/very hard (n=10) | Easy/very easy (n=148) | Total (N=158a) | ||
|
|||||
|
Minimum to maximum | −1.10 to 3.00 | −2.00 to 3.20 | −2.00 to 3.20 |
|
|
Median (Q1-Q3) | −0.30 (−1.00 to 0.20) | 0.70 (0.00 to 1.20) | 0.60 (−0.10 to 1.20) | .02 |
|
|
||||
|
Minimum to maximum | 44 to 77 | 44 to 79 | 44 to 79 |
|
|
Median (Q1 to Q3) | 49.5 (48.0 to 52.0) | 54.5 (49.5 to 61.0) | 54.0 (49.0 to 61.0) | .16 |
aOne participant skipped the feasibility question.
bWilcoxon test.
Participants who thought the web-based survey was easy or very easy to complete were more likely to prefer to answer future questions via computer or tablet than those who found it hard or very hard (
Comparisons of evaluable responses by questionnaire administration are summarized in
Evaluable responses by questionnaire format (N=159).
Question | Paper and pencil: proportion of evaluable responses | Web-based survey: proportion of evaluable responses | ||||
|
n (%) | N | n (%) | N |
|
|
|
||||||
|
Any sexual partners | 140 (88.1) | 159 | 155 (97.5) | 159 | .001 |
|
Number of sexual partners | 67 (73) | 92 | 75 (90) | 83 | .003 |
|
Oral sex with a man | 81 (88) | 92 | 65 (71) | 92 | .004 |
|
Oral sex with a man with condom | 59 (82) | 72 | 57 (90) | 63 | .15 |
|
Oral sex with a woman | 73 (79) | 92 | 77 (93) | 83 | .01 |
|
Oral sex with a woman with condom | 8 (28) | 29 | 7 (54) | 13 | .10 |
|
Vaginal sex | 75 (82) | 92 | 78 (94) | 83 | .01 |
|
Vaginal sex with condom | 22 (54) | 41 | 23 (79) | 29 | .03 |
|
Anal sex with a man | 76 (83) | 92 | 79 (95) | 83 | .009 |
|
Anal sex with a man with condom | 40 (69) | 58 | 41 (91) | 45 | .007 |
|
Anal sex with a woman | 69 (75) | 92 | 79 (95) | 83 | <.001 |
|
Anal sex with a woman with condom | 1 (4) | 24 | 3 (43) | 7 | .007 |
|
Any new sexual partners | 83 (90) | 92 | 79 (95) | 83 | .21 |
|
Number of partners who know your HIV status | 84 (91) | 92 | 75 (90) | 83 | .83 |
|
Number of partners with known HIV status | 86 (93) | 92 | 77 (93) | 83 | .85 |
|
||||||
|
How often drink alcohol | 159 (100.0) | 159 | 158 (99.4) | 159 | .32 |
|
How many drinks containing alcohol | 99 (98.0) | 101 | 96 (96.0) | 100 | .40 |
|
How often binge drink alcohol | 101 (100.0) | 101 | 99 (99.0) | 100 | .31 |
|
Not getting things done because of alcohol | 98 (97.0) | 101 | 99 (99.0) | 100 | .32 |
|
Emotional problems from alcohol | 96 (95.0) | 101 | 100 (100.0) | 100 | .02 |
|
Last time used tobacco | 156 (98.1) | 159 | 157 (98.7) | 159 | .65 |
|
Last time used marijuana | 157 (98.7) | 159 | 158 (99.4) | 159 | .56 |
|
Last time used cocaine | 157 (98.7) | 159 | 158 (99.4) | 159 | .56 |
|
Last time used heroin | 156 (98.1) | 159 | 159 (100.0) | 159 | .08 |
|
Last time used amphetamines | 154 (96.9) | 159 | 159 (100.0) | 159 | .02 |
|
Last time used other nonprescribed substance | 152 (95.6) | 159 | 159 (100.0) | 159 | .007 |
|
Not getting things done because of substance use | 86 (97) | 89 | 98 (98.0) | 100 | .56 |
|
Emotional problems from substance use | 85 (96) | 89 | 98 (98.0) | 100 | .33 |
|
||||||
|
How many days spent doing vigorous activities | 153 (96.2) | 159 | 159 (100.0) | 159 | .01 |
|
How much time spent doing vigorous activities | 79 (90) | 88 | 85 (98) | 87 | .03 |
|
How many days spent doing moderate activities | 149 (93.7) | 159 | 159 (100.0) | 159 | .001 |
|
How much time spent doing moderate activities | 91 (82.7) | 110 | 102 (99.0) | 103 | <.001 |
|
How many days spent walking ≥10 min | 148 (93.1) | 159 | 159 (100.0) | 159 | <.001 |
|
How much time spent walking | 119 (87.5) | 136 | 140 (99.3) | 141 | <.001 |
|
How much time spent sitting | 126 (79.2) | 159 | 156 (98.1) | 159 | <.001 |
To our knowledge, this is one of the first studies on the feasibility and acceptability of web-based data collection for clinical research purposes among older adults living with HIV, and our findings indicate that web-based surveys can successfully be implemented in research with this population. The perceived acceptability and feasibility of the web-based survey were high, and almost all participants found the survey to be easy or very easy to complete. The data quality of responses via the web-based survey was similar to or higher than that for the same questions in the paper-and-pencil questionnaires.
Although overall acceptability and feasibility were high, there were differences by demographic and clinical factors that need to be taken into account when deciding how to incorporate web-based data collection. Younger adults were more likely than older adults to prefer a web-based format. Although the adoption of web-based technologies has steadily increased in recent years among older adults in the United States, it continues to lag behind those of younger adults, as does confidence in one’s ability to use these technologies [
Although data quality within the web-based format was not affected by either age or neurocognitive function, the fact that participant preference did depend on these factors has to be considered when determining how to incorporate web-based surveys in a clinical research study. A mixed mode study design is likely needed, with participants given the option of either web-based or paper-and-pencil administration [
Web-based survey acceptability was not affected by the presence of frailty or health comorbidities. Frailty was also not associated with preference or interest in technology use in a previous study of older adults [
The findings of this pilot study indirectly inform a longer-term goal of incorporating remotely completed web-based assessments into A5322. In addition to age- or cognitive-based limitations with completing web-based assessments, remote completion also requires internet/smartphone access at home as well as the skills needed to independently access and submit web-based surveys. Given these caveats, the option of remote data completion may increase retention among participants who find it difficult to maintain regular clinic visits because of time, travel, or health restrictions. Although studies that rely solely on remote data collection are likely to have high attrition rates [
This pilot study was limited to participants whose primary language was English, and the design required that the web-based questionnaires be completed on site. Therefore, the results might not be generalizable to non–English-speaking individuals or to those without adequate computer or tablet or internet access. We were unable to have participants complete the web-based and paper-and-pencil questionnaires at the same time and therefore were not able to compare frequencies of behaviors by format because we could not assume that the frequency of these behaviors would be stable over time. We did not take within-person correlations into account when comparing the proportion of evaluable responses across survey administration types. Many of the questions were designed to be answered only by participants who endorsed a leading question that triggered another question (eg, the leading question “in the past 6 months, have you had sex with another person?” would trigger subsequent questions on the type of partners and condom use). The number of participants answering each question differed by mode of administration because they were asked at different points in time, with different frequencies of behaviors reported. Therefore, a matched approach was not possible for all comparisons. However, we were able to use McNemar test for a subset of questions (those asked of all participants) to account for within-person correlations. In these situations, the McNemar
We found that in a group of older adults living with HIV being followed in a longitudinal observational study, completion of a web-based survey of questionnaires assessing sexual, substance use, and physical activity behaviors was perceived to be highly acceptable and feasible, with data quality on average being higher with the web-based versus paper-and-pencil format. As persons living with HIV continue to age, often with comorbidities and disabilities, long-term participation in research will become more challenging, even as the need to understand their health-related outcomes continues to grow [
Web-based survey questions with ≥1 rather not answer response.
Rather not answer responses by acceptability and feasibility.
Evaluable responses by questionnaire format (comparing web-based surveys with paper-and-pencil questionnaires from all A5322 participants).
Comparison of evaluable responses—chi-square test versus McNemar test.
AIDS Clinical Trials Group
Instrumental Activities of Daily Living
National Institutes of Health
odds ratio
Research reported in this publication was supported by the National Institute of Allergy and Infectious Diseases of the National Institutes of Health (NIH) under Award Number UM1 AI068634, UM1 AI068636, and UM1 AI106701. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. The authors would like to thank the participants and study staff for their commitment and participation.
CF has received research grant support via his institution from Janssen Pharmaceuticals, Merck & Co, ViiV Healthcare, Gilead Sciences, and Amgen Inc.