We conducted a cross-sectional study using an anonymous online survey. We recruited respondents between March 14, 2019, and April 8, 2019, via Amazon’s Mechanical Turk (MTurk) platform. MTurk is an online crowdsourcing tool that recruits anonymous users to participate in a variety of computer-based tasks, including assessment of web content [26]. Participants were eligible to respond if they were over 18 years of age, English speaking, and given the “Masters Qualification” by MTurk. Amazon designates participants as “Masters” if they demonstrate reliability in completing a large number and variety of tasks posted by different requesters [27]. We advertised the survey task as “Public Opinion about Abortion Websites.” We compensated participants US $2.50 to complete the assignment. After the first 425 responses, we increased compensation to US $3.00 for the final 75 responses in order to increase recruitment. Participants were limited to responding to the survey once based on their unique worker ID number that is assigned by Amazon and is attached to their taxpayer number [27]. The study was reviewed and approved by the Oregon Health & Science University institutional review board.

We selected our websites from a database of abortion websites used in a previous study [10]. For each website in that database, three investigators determined the slant (prochoice, neutral, or antichoice) and slant clarity (easy, moderate, or difficult to determine website slant) based on their impression of the website. All three researchers have published abortion-related research. We then selected 11 websites categorized as both “antichoice” and “difficult to determine” in slant (Multimedia Appendix 1). These 11 antichoice websites included one state government website (Alaska Department of Health and Social Services), four CPC pages associated with brick and mortar establishments, and six websites that detail abortion risks and side effects. We included one website that was rated as being “neutral” in slant and “easy to determine” in slant clarity (The Louisiana Department of Public Health website on “Methods & Medical Risks”). We selected the Louisiana website as a comparator for the antichoice websites.

We used previously validated studies measuring website trust [17-20,24,25] to develop our survey (Table 1). We included items across the following four domains: creators, design, content, and overall trust (Table 1). Participants viewed each website and responded to statements about website characteristics using a 7-point Likert scale ranging from −3 (“strongly disagree”) to 3 (“strongly agree”) [20,21,25,28]. We assigned each participant to assess three randomly selected antichoice websites and the Louisiana website. Therefore, all participants viewed the Louisiana website but only a subset of participants viewed each of the 11 antichoice websites. Our survey displayed all four websites in random order, and users had 18 minutes to complete the survey and rate all four websites.

We compiled our primary outcome (“mean overall trust”) by averaging three of the four items comprising the overall trust domain. During analysis, we excluded one item “this website is biased” from the primary outcome owing to low correlation with the other questions in the trust domain. We reverse coded responses to items 14 “this website contains errors” and 20 “this website is biased,” so that all positive Likert scale responses become markers of a favorable characteristic.

After participants evaluated all four websites (responses were not required to move on to the next question), we asked sociodemographic questions, including questions on age, race/ethnicity, gender, educational background, state of residence, residential density (rural, urban, and suburban), and internet use (hours per week online) [29]. Unfortunately, an error in the MTurk survey assigned Alabama to those who did not respond to the item regarding state of residence. Finally, we asked about views on abortion, including whether abortion should be legal (no law limiting abortion, some limits, severe limits, or should be illegal in all circumstances) and self-rated abortion knowledge (1-10, with 1 being no knowledge and 10 being expert knowledge).

We used RStudio Version 1.2.1335 for all analyses [30]. We used descriptive statistics to characterize the overall demographics of our participants. Due to small numbers, we collapsed race and ethnicity into a binary category (white vs not white). We then conducted analyses for individual items and domains at the website level and multivariate analysis to determine relationships between demographic variables and overall trust.

For each website, we calculated mean scores for responses to the individual items as well as composite means for the four domains, including the primary outcome of mean overall trust. We tested bivariate associations between mean overall trust and each survey item from Table 1. We used correlation coefficients to measure the strength of the relationship of each individual item in the survey with the items comprising the “overall trust” category (info trust, quality, and recommend) as well as mean overall trust itself. Correlation coefficient scores range from “1” (perfect positive correlation) to “−1” (perfect negative correlation), with “0” meaning there is no correlation between two items. In order to test the robustness of our findings, we assessed these correlations in the following three ways: for all 12 websites combined, for the antichoice websites combined, and for the prochoice Louisiana website alone. We set 0.5 as the cutoff for a “marked” correlation and considered coefficients between 0.4 and 0.5 as indicating “medium” correlation [31].

Last, we used linear regression to identify participant characteristics associated with mean overall trust, where mean overall trust was treated as a continuous variable. We excluded participants with missing items in the trust domain from multivariate modeling (36/498). We created two models. One model was created for the antichoice websites, and another for the Louisiana website alone in order to compare model findings stratified by slant. Variables in our model included race, age, self-rated abortion knowledge, abortion views, hours spent on the internet per week, educational attainment, and residential density. We selected these covariates based on bivariate relationships and prior literature suggesting that these variables impact website trust [24].

We obtained surveys from 500 participants and excluded only two respondents who did not complete 50% of the website survey questions. We included all other partial responses. Of the 498 respondents, 403 (80.9%) completed all questions for all four websites. Of the 95 incomplete surveys, 53 (56%) missed only one item on one website and fully completed the other three website evaluations. An additional 20 (21%) missed two or three items total across the four websites. A total of 6 (6%) participants missed all 22 items from one website but completed all items from the other three websites.

Full demographic characteristics are shown in Table 2. Respondent age ranged from 22 to 70 years, and 50.1% (247/493) identified as female. Just over half (279/495, 56.3%) of the respondents attained at least a bachelor’s degree. The majority of our respondents were white (327/495, 66.1%), and an additional 20% identified as Asian. Participants were distributed across the US census region, with 16.7% (83/498) from the Northeast, 34.7% (173/498) from the South, 19.5% (97/498) from the Midwest, 20.1% (100/498) from the West, and 9.1% (45/498) not from the United States (Table 2). The MTurk error makes it likely that 34.7% is higher than the true proportion of respondents who are from the South. On average, respondents reported spending 35.6 hours per week on the internet (95% CI 1.0-72.8). The majority of participants self-identified as prochoice, with 80.3% (399/497) indicating that there should only be some limits or no law limiting access to abortion. We found self-reported abortion knowledge in our participants to be normally distributed with a mean of 5.9/10 (95% CI 2.4-9.4).

Mean scores were calculated for each of the four domains (content, creators, design, and overall trust). The mean score of the overall trust domain is the primary outcome of mean overall trust. Domain scores ranged from −0.66 to 1.89, where −3 is highly unfavorable and 3 is highly favorable (Table 3). For the primary outcome of mean overall trust, the Louisiana State Health website had the highest score (1.89) and smallest standard deviation (1.27) (Table 3). In addition, the Louisiana State Health website had the lowest standard deviations in the content (1.27) and design (1.65) domains. By comparison, the Alaska State Health website received the highest score among all websites for creators (1.41), a high score for mean overall trust (1.63), and lower scores for content (1.05) and design (0.97) (Table 3).

Eight of the 11 antichoice websites were associated with a positive mean overall trust score (Table 3). Abortion Facts, Abortion Pill Risks, and U Pregnancy received negative scores for mean overall trust, ranging from −0.66 to −0.04. In addition, Baby Gaga received a score of −0.15 in the creators domain and U Pregnancy received scores of −0.14 for the content domain, −0.54 for the creators domain, and −0.23 for the design domain.

Pooled correlation coefficient matrices were used to compare each item to the three items included in the primary outcome of mean overall trust as well as to the primary outcome of mean overall trust (Table 4, Table 5, and Table 6). We assessed if the correlation between website characteristics and mean overall trust was different for the antichoice websites and the neutral Louisiana website. For both the antichoice websites and the Louisiana website, the creator confidence item (“I believe that the creators of this website are honest and trustworthy”) correlated most highly with mean overall trust (Table 5 and Table 6). For the Louisiana website, we found that the creator confidence item was the only item markedly correlated with mean overall trust (coefficient=0.66) (Table 6). In contrast, for the antichoice websites, professional appearance (coefficient=0.59), look and feel (coefficient=0.59), perception that the information is created by experts (coefficient=0.59), association with a trustworthy organization (coefficient=0.58), valued features and functionalities (coefficient=0.54), and interactive capabilities (coefficient=0.52) all demonstrated marked relationships with mean overall trust. Up-to-date appearance (coefficient=0.50), overall design (coefficient=0.49), ability to navigate the website (coefficient=0.48), and presence of images (coefficient=0.46) had medium correlation with mean overall trust. We found no correlation between presence of spelling or grammar errors and mean overall trust (coefficient=−0.07) (Table 5).

We performed multivariate analysis to identify associations between participant demographics and the primary outcome of mean overall trust. We found that preference for fewer abortion restrictions was associated with higher mean overall trust of the Louisiana website (B=−0.43, 95% CI −0.87 to 0.01) and with lower mean overall trust of the antichoice websites (B=0.52, 95% CI 0.05-0.99) (Table 7 and Table 8). In addition, white race was associated with lower mean overall trust of the antichoice websites (B=−1.41, 95% CI −2.26 to −0.55) (Table 8). Other participant characteristics were not associated with mean overall trust.