We conducted a 2×2 factorial design randomized controlled trial (RCT) to test 2 recruitment strategies for the invitation email in a Web-based trial with participants who were members of the Japan Primary Care Association (JPCA).

This study was a substudy regarding the recruitment process embedded in another randomized controlled study (Do Overstated Conclusions Trick Our Readers? The DOCTOR study [UMIN000025317]), which investigated whether overstated conclusions in abstracts of clinical research papers can bias primary care doctors’ impressions of the results. The DOCTOR study was conducted between January 17, 2017, and February 8, 2017. In the DOCTOR study, participants were asked to complete a 2-page online questionnaire about evidence-based medicine. The questionnaire had 9 and 4 items, respectively, on the first and second pages. A completeness check was automatically conducted after the questionnaire was submitted and respondents were asked to complete the mandatory items if they were blank. The first researcher (RS) developed all the Web systems for the DOCTOR study. Details of the methods and results of the DOCTOR study are reported elsewhere [19].

To maximize the number of respondents in the DOCTOR study, we recruited the participants in 2 phases. In the first phase, we randomly sampled 2000 primary care physicians from 7040 potential participants and sent the invitation email to investigate the most effective subject line (emphasizing an incentive of a chance to win an Amazon gift card worth 3000 Japanese yen [US $30]; Amazon gift cards were to be provided to 20 respondents through a lottery) on different days of the week (Tuesday vs Friday). In the second phase, we invited remaining potential participants using the most effective method confirmed in the first phase. We hereby report the results from the first phase. The English version of the invitation email is included in the supplementary appendix reported elsewhere [19].

We tested the 2 interventions related to the invitation email. One intervention involved mentioning the chance to receive an Amazon gift card in the subject line (“[Get Amazon gift card] Survey on EBM for 5 minutes”) or not (“[Funded by the association] Survey on EBM for 5 minutes”). The other intervention involved sending the invitation email at 6:00 PM on either Tuesday, January 17, 2017, or Friday, January 20, 2017.

The eligibility criteria for participation were as follows: being a member of the JPCA, which is the largest association of primary care doctors in Japan; having more than 2 years of clinical practice experience; and having a registered email address with the JPCA. Eligible participants of the DOCTOR study were 7040 primary care doctors out of the 10,851 members of the JPCA in January 2017 when the DOCTOR study was conducted.

To limit participants to members of the JPCA, we did not announce the DOCTOR study on websites or social networking services with open access. The recruitment process was conducted only through emails from the secretariat of the JPCA, which invited doctors to voluntarily participate in the study.

We randomly assigned a sample of 2000 primary care physicians who were randomly selected from the eligible JPCA members to 4 groups (500 participants in each group): (1) those who received the invitation email with a subject line emphasizing the incentive on Tuesday, (2) those who received the invitation email with a subject line emphasizing the incentive on Friday, (3) those who received the invitation email with a subject line not emphasizing the incentive on Tuesday, and (4) those who received the invitation email with a subject line not emphasizing the incentive on Friday.

The invitation emails were sent by the secretariat of the JPCA according to the allocation table prepared by RS using a computer-generated random sequence. Potential participants were not informed that the invitation emails were sent in 4 different ways. Outcomes such as response, completion, and access to the DOCTOR study website were automatically recorded.

The primary outcome was the response rate defined as the number of participants completing the first page divided by the number of invitation emails delivered. The formula was equivalent to Response Rate 2, defined by the American Association for Public Opinion Research [20], which was used in a previous study investigating strategies to improve response rates for a Web-based clinician study [21].

Secondary outcomes were the completion rate and access rate to the DOCTOR study website. The completion rate was defined as the number of respondents who completed all the pages on the DOCTOR study website divided by the number of invitation emails delivered. The access rate was the number of participants accessing the DOCTOR study website divided by the number of invitation emails delivered.

The data were collected between January 17, 2017, and February 8, 2017. The response rate and completion rate were recorded by a Web application developed by RS for the DOCTOR study. The access rate was recorded using Google Analytics (Google). The allocation of each participant was tracked using a URL link unique to each allocation embedded in the invitation email. We used an IP address and a cookie to avoid double-counting the same user if any of them accessed or responded more than once. No bug fixes, downtime, or content changes were necessary during the period of the DOCTOR study.

We included 2000 of the potential participants of the DOCTOR study in this factorial RCT to apply the most effective invitation email strategy to the remaining potential participants. With a sample size of 2000, we calculated the power to be 0.81 at an alpha level of .05 to detect the significant absolute risk difference of 3% against an assumed control response rate of 4% based on results of previous surveys targeting JPCA members.

Statistical analyses were performed using R version 3.3.1 (The R Foundation). To compare the response rates, completion rates, and access rates between groups, we reported both risk ratios and risk differences with 95% confidence intervals using the chi-squared test. We also compared demographics of respondents between the intervention groups. We used the chi-squared test for dichotomous variables and Student t test for continuous variables. We set the threshold of statistical significance at P ≤.05. All analyses were conducted according to the intention-to-treat principle.

The DOCTOR study was approved by the Institutional Review Board (IRB) of Kyoto University Graduate School of Medicine. We did not include the present factorial RCT in the DOCTOR study when applying for an ethics review by the IRB because the aims of the study were not directly related to health care. The Board of Directors of the JPCA approved this factorial RCT in the recruitment processes of the DOCTOR study. The participants of the DOCTOR study provided their consent on the website.

To avoid the risk of personal information leak, we did not obtain identifiable personal information from participants other than winners of the incentive lottery. In addition, identifiable personal information of the winners was stored separately from the research data.

The DOCTOR study was registered prospectively at University Hospital Medical Information Network Clinical Trials Registry [UMIN000025317].

Figure 1 shows the participant flow. Out of 2000 invitation emails, 57 were undeliverable; therefore, we analyzed data from the remaining 1943 participants for all outcomes. Of 1943 primary care doctors, 148 (7.61%) accessed the study site, 123 (6.33%) responded, and 118 (6.07%) completed the questionnaire.

Table 1 shows the characteristics of the 123 respondents. We did not find statistically significant differences in the characteristics among the 4 groups except for primary workplace at the nominal P value of .01 without adjusting for multiple comparisons.

Access, response, and completion rates in each randomized group (Tuesday or Friday × incentive in the subject line or not) are reported with 95% confidence intervals in Table 2. We found no statistically significant difference in all the outcomes among the 4 groups. The interaction effects on all the outcomes between the 2 factors were not statistically significant. In the following sections, we therefore report the effect of the 2 interventions separately.

Table 3 summarizes the rates of access, response, and completion. The response rates, which were the primary outcome of this study, were 6.7% (65/972) in the intervention group and 6.0% (58/971) in the control group, respectively. There was no significant difference in the response rate between the 2 groups: the risk ratio was 1.12 (95% CI 0.80 to 1.58) and the risk difference was 0.7% (95% CI –1.5% to 2.9%). Similarly, the completion rate and access rate were almost the same in the 2 groups (see Table 3).

Table 4 shows all outcomes regarding the day of the week of sending the invitation email. The response rate was 6.3% (61/972) on Tuesday evening and 6.4% (62/971) on Friday evening.

No significant difference was demonstrated between the Tuesday group and Friday group: the risk ratio was 0.98 (95% CI 0.70 to 1.38) and the risk difference was –0.1% (95% CI –2.3% to 2.1%).

This study was the first to investigate the effect of emphasizing the incentive in the subject line of the invitation email and the day of the week of sending the invitation email on physicians’ participation in a Web-based study. The cumulative response rate was 6.3% (123/1943). We found that emphasizing the incentive in the subject line of the invitation email did not significantly improve outcomes such as the response rate, completion rate, or access rate. Likewise, sending the invitation email on Tuesday or Friday did not affect the outcomes.

Previous studies reported conflicting results related to the effect of emphasizing incentives in the subject line on response rates [12-14]. In studies that found a significant effect of the subject line on the response rate of Web-based studies, the magnitude of the effect was about 1.1 to 1.2 in the risk ratio [8,13]. Thus, one possible explanation is that our study was underpowered due to the very low control response rate. There is, however, less practical meaning in the risk ratio of 1.1 in the situation of the expected response rate of 4%, such as in this study. Our sample size was large enough to detect the practically meaningful difference in the risk ratio of 1.6 to 1.7. The second possible reason is that the incentive mentioned in the subject line was not attractive enough to motivate primary physicians to respond to the invitation. Our subject line mentioned a drawing for a US $30 Amazon gift card, which might be less attractive than incentives paid in cash or by check which can be used anywhere. Furthermore, lottery incentives were reported to be less effective than fixed incentives [22]. The third possible reason is the effect of the control subject line mentioning the support from the JPCA. The authority effect might have reduced the between-group difference in outcomes with regard to the subject line.

Regarding the day of the week of sending the invitation email, our results were consistent with those of previous studies concerning postal invitation. Sending the invitation email on Tuesday evening or Friday evening did not affect the study participation of the primary care doctors in the Web-based study. Pit et al [15] synthesized the results of 3 studies and reported no significant difference in doctors’ participation in paper-based studies based on the day of the week on which the postal mails were sent or received. Our results suggest that the day of the week an invitation email is sent does not increase doctors’ participation in Web-based studies, similar to postal mail invitations for paper-based studies.

The 2 interventions about invitation emails failed to trigger primary care physicians’ response. The cumulative response rate of 6.3% is low, but it is not uncommon in Web-based studies for physicians [1]. Our results suggest that using additional media such as postal mail, telephone, or social media may be needed to attain a large enough response rate [23,24].

There are several limitations to this study. One is its generalizability. All the participants were primary care physicians in Japan; therefore, it is unclear whether our findings can be applied to those in other countries. Furthermore, primary care physicians, who were the target population of this study, are known to respond to surveys less than physicians in other specialties [25]. Our results, therefore, may not be generalizable to physicians in other specialties. There were additional limitations related to the interventions. In terms of subject lines, we showed that mentioning a lottery for an Amazon gift card (US $30) as an incentive was not effective. However, we could not clarify the effect of subject lines emphasizing larger monetary incentives paid in cash or by check than ours, as $30 may not have been considered attractive enough by Japanese physicians. As for the effect of the sending day, we only compared Tuesday to Friday; therefore, the effect of other days of the week remains unclear.

However, there are also several strengths to this study. This is the first study to investigate the effect of 2 invitation email strategies targeting physicians. As participation rates in Web-based surveys for physicians are decreasing by year, our findings would provide important information when designing Web-based surveys, especially for primary care physicians. In addition, we used a rigorous RCT design, which avoided the risk of bias as much as possible. Furthermore, our sample size was large enough to detect a practically meaningful difference between the groups.

Neither emphasizing a monetary incentive in the subject line of the invitation email nor varying the sending day of the invitation email increased the response rate in a Web-based physician study. Further studies are needed to find an effective invitational strategy using multimodal media beside email to improve response rates of physicians in Web-based studies.