The Impact of Web-Based Ratings on Patient Choice of a Primary Care Physician Versus a Specialist: Randomized Controlled Experiment

Background Physician review websites have empowered prospective patients to acquire information about physicians. However, little is known about how Web-based ratings on different aspects of a physician may affect patients’ selection of physicians differently. Objective The objectives of this study were to examine (1) how patients weigh ratings on a physician’s technical skills and interpersonal skills in their selection of physicians and (2) whether and how people’s choice of a primary care physician versus a specialist is affected differently by Web-based ratings. Methods A 2×2×2×2 between-subjects experiment was conducted. Over 600 participants were recruited through a crowdsourcing website and randomly assigned to view a mockup physician review Web page that contained information on a physician’s basic information and patients’ ratings. After reviewing the Web page, participants were asked to complete a survey on their perceptions of the physician and willingness to seek health care from the physician. Results The results showed that participants were more willing to choose a physician with higher ratings on technical skills than on interpersonal skills compared with a physician with higher ratings on interpersonal skills than on technical skills, t369.96=22.36, P<.001, Cohen d=1.22. In the selection of different types of physicians, patients were more likely to choose a specialist with higher ratings on technical skills than on interpersonal skills, compared with a primary care physician with the same ratings, F1,521=5.34, P=.021. Conclusions The findings suggest that people place more weight on technical skills than interpersonal skills in their selection of a physician based on their ratings on the Web. Specifically, people are more likely to make a compromise on interpersonal skills in their choice of a specialist compared with a primary care physician. This study emphasizes the importance of examining Web-based physician ratings in a more nuanced way in relation to the selection of different types of physicians. Trial Registration ISRCTN Registry ISRCTN91316463; http://www.isrctn.com/ISRCTN91316463

Is this a full powered effectiveness trial or a pilot/feasibility trial? * Manuscript tracking number * If this is a JMIR submission, please provide the manuscript tracking number under "other" (The ms tracking number can be found in the submission acknowledgement email, or when you login as author in JMIR. If the paper is already published in JMIR, then the ms tracking number is the four-digit number at the end of the DOI, to be found at the bottom of each published article in JMIR) 1a-i) Identify the mode of delivery in the title Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internet-based" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if o ine products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms.
Does your paper address subitem 1a-i? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Web-Based Ratings" 1a-ii) Non-web-based components or important co-interventions in title Mention non-web-based components or important co-interventions in title, if any (e.g., "with telephone support").
Does your paper address subitem 1a-ii?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Patient Choice of a Primary Care Physician Versus a Specialist" 1a-iii) Primary condition or target group in the title Mention primary condition or target group in the title, if any (e.g., "for children with Type I Diabetes") Example: A Web-based and Mobile Intervention with Telephone Support for Children with Type I Diabetes: Randomized Controlled Trial Does your paper address subitem 1a-iii? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Patient choice" 1b-i) Key features/functionalities/components of the intervention and comparator in the METHODS section of the ABSTRACT Mention key features/functionalities/components of the intervention and comparator in the abstract. If possible, also mention theories and principles used for designing the site. Keep in mind the needs of systematic reviewers and indexers by including important synonyms. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) Does your paper address subitem 1b-i? * Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "A 2x2x2x2 between-subjects experiment was conducted. Over 600 participants were recruited through a crowdsourcing website and randomly assigned to view a mockup physician review Web page that contained information on a physician's basic information and patients' ratings. After reviewing the Web page, participants were asked to complete a survey on their perceptions of the physician and willingness to seek health care from the physician. " Does your paper address subitem 1b-ii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study fully automated 1b-iii) Open vs. closed, web-based (self-assessment) vs. face-toface assessments in the METHODS section of the ABSTRACT Mention how participants were recruited (online vs. o ine), e.g., from an open access website or from a clinic or a closed online user group (closed usergroup trial), and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment). Clearly say if outcomes were self-assessed through questionnaires (as common in web-based trials). Note: In traditional o ine trials, an open trial (open-label trial) is a type of clinical trial in which both the researchers and participants know which treatment is being administered. To avoid confusion, use "blinded" or "unblinded" to indicated the level of blinding instead of "open", as "open" in web-based trials usually refers to "open access" (i.e. participants can self-enrol). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) Does your paper address subitem 1b-iii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Over 600 participants were recruited through a crowdsourcing website and randomly assigned to view a mockup physician review Web page that contained information on a physician's basic information and patients' ratings. After reviewing the Web page, participants were asked to complete a survey on their perceptions of the physician and willingness to seek health care from the physician. " 1b-iv) RESULTS section in abstract must contain use data Report number of participants enrolled/assessed in each group, the use/uptake of the intervention (e.g., attrition/adherence metrics, use over time, number of logins etc.), in addition to primary/secondary outcomes. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it)

INTRODUCTION
Does your paper address subitem 1b-iv?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The results showed that participants were more willing to choose a physician with higher ratings on technical skills than on interpersonal skills compared with a physician with higher ratings on interpersonal skills than on technical skills, t369.96=22.36, P<.001, Cohen d=1.22. In the selection of different types of physicians, patients were more likely to choose a specialist with higher ratings on technical skills than on interpersonal skills, compared with a primary care physician with the same ratings, F1,521=5.34, P=.021. " 1b-v) CONCLUSIONS/DISCUSSION in abstract for negative trials Conclusions/Discussions in abstract for negative trials: Discuss the primary outcome -if the trial is negative (primary outcome not changed), and the intervention was not used, discuss whether negative results are attributable to lack of uptake and discuss reasons. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) Does your paper address subitem 1b-v?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The ndings suggest that people place more weight on technical skills than interpersonal skills in their selection of a physician based on their ratings on the Web. Speci cally, people are more likely to make a compromise on interpersonal skills in their choice of a specialist compared with a primary care physician. This study emphasizes the importance of examining Web-based physician ratings in a more nuanced way in relation to the selection of different types of physicians." Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "With the growing popularity of PRWs, researchers have recently begun to examine the role of these websites in people's health decision making [5][6][7][8]. The scholarship on PRWs covers a wide range of topics, including but not limited to demographics of website users, structures of the portals, patterns of website usage, and content of reviews [9][10][11]. Technical skills and interpersonal skills reside at the core of a physician's quali cations and are commonly rated on PRWs [12,13]. However, little is known about how ratings on these different aspects of a physician may affect patients' choice differently [14]. Previous research presents mixed results on how people set the priority of technical and interpersonal skills in physician selection [15,16]. Therefore, the rst goal of this study was to examine how patients prioritize technical and interpersonal skills in their physician selection based on ratings on PRWs.
The second objective of this study was to examine whether and how people's choice of a primary care physician versus a specialist is affected differently by Web-based ratings. METHODS 3a) Description of trial design (such as parallel, factorial) including allocation ratio 3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons Does your paper address CONSORT subitem 2b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "RQ1: Are people more willing to choose a physician with higher ratings on technical skills than on interpersonal skills, or a physician with higher ratings on interpersonal skills than on technical skills? H1: People are more willing to choose a specialist who has higher ratings on technical skills than on interpersonal skills, compared with a primary care physician who has the same ratings.
H2: People are more willing to choose a primary care physician who has higher ratings on interpersonal skills than on technical skills, compared with a specialist who has the same ratings." Does your paper address CONSORT subitem 3a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "To investigate the proposed research question and hypotheses, a 2 (ratings on subitem not at all important 1 2 3 4 5 essential 4a) Eligibility criteria for participants Does your paper address CONSORT subitem 3b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study No changes to methods were made after trail commencement.
3b-i) Bug xes, Downtimes, Content Changes Bug xes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug xes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have in uenced study design such as staff changes, system failures/downtimes, etc. [2].
Does your paper address subitem 3b-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study No changes were made afterwards.
Does your paper address CONSORT subitem 4a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "A total of 608 participants completed the Web-based experiment. Participants were recruited via the crowdsourcing website, Amazon's Mechanical Turk (mTurk), and compensated for their time. We excluded people who failed the attention checks (n=26) and those who spent no time (n=1) or less than 5 seconds on the Web page (n=41). " Computer / Internet literacy is often an implicit "de facto" eligibility criterion -this should be explicitly clari ed.
Does your paper address subitem 4a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Participants' use of physician review websites was measured and entered as a control variable.
4a-ii) Open vs. closed, web-based vs. face-to-face assessments: Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. o ine), e.g., from an open access website or from a clinic, and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email con rmation, phone calls) were used to detect/prevent these.
Does your paper address subitem 4a-ii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Participants were anonymous in this online-only trial. Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results.
Does your paper address subitem 4a-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Each participant read and sign the consent form online before conducting the study.
Does your paper address CONSORT subitem 4b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Data were collected online. Participants completed the study using a digital device/at a location of their choice.
4b-i) Report if outcomes were (self-)assessed through online questionnaires Clearly report if outcomes were (self-)assessed through online questionnaires (as common in webbased trials) or otherwise. Does your paper address subitem 4b-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Institutional a liations were not displayed in the stimuli or survey.

5-i) Mention names, credential, a liations of the developers, sponsors, and owners
Mention names, credential, a liations of the developers, sponsors, and owners [6] (if authors/evaluators are owners or developer of the software, this needs to be declared in a "Con ict of interest" section or mentioned elsewhere in the manuscript).
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Following consent, participants were presented a cover story to read. On the basis of the type of physician that they were assigned to, participants were asked to imagine themselves in a situation looking for either a primary care physician or a surgeon. The vignette about a primary care physician depicted a situation that the participant recently moved to a new city and was in need of a new primary care physician. Owing to a lack of input from family members and friends, they decided to search for primary care physicians on PRWs. The vignette about a surgeon described a situation in which the participant had lasting back pains. The primary care physician suspected that the patient may need spinal surgery and provided a list of surgeons to choose from. The participant decided to search for the recommended surgeons on PRWs. After reading through the scenario and imaging themselves in the described situation, each participant was directed to a physician review page to learn about the physician.
A total of 16 physician review pages were developed for this study (see Figure 1). The top part of each page listed basic information about a physician, including the physician's name (Dr J Smith), the specialty (family medicine or surgeon), and information on new patient acceptance (accepting new patients). To manipulate the type of a physician, half of the Web pages listed the physician's specialty as family medicine and the other half described the physician as a surgeon.
Each page contained 4 aggregated rating categories about Dr Smith, including 2 items on technical skills ("My doctor accurately diagnosed my problem" and "My doctor effectively treated my problem") and 2 on interpersonal skills ("My doctor was caring" and "My doctor spent enough time with me"). Past research has suggested that a physician's skills on diagnosis and treatment are among the most important considerations when selecting a physician [32]. In addition, a physician's personal manner as well as time spent with a patient are critical to a patient's satisfaction on the physician's interpersonal skills [32,33]. These 4 categories frequently appear on PRWs [34] and thus are adopted in this study. To manipulate the valence of physician ratings, these rating categories were assigned different star ratings. Each rating category was presented in the form of aggregated ratings. In the conditions where a physician received high ratings on technical skills, the 2 items pertaining to technical skills were given 5/5-star ratings. In the conditions of moderate ratings on technical skills, the same items were assigned 3/5-star ratings. We chose to examine moderate ratings instead of low ratings in this study because h t th t l ti l ti l PRW Th l essential research suggests that low ratings are relatively uncommon on PRWs . The valence of a physician's interpersonal skills was manipulated in the same way. Furthermore, the rating categories were presented to participants in counterbalanced order to control for the impact of rating order effects. In half of the experimental conditions, the 2 rating categories on technical skills were displayed before the 2 categories on interpersonal skills. In the other half, ratings on technical skills were presented beneath the ratings on interpersonal skills. "

5-ii) Describe the history/development process
Describe the history/development process of the application and previous formative evaluations (e.g., focus groups, usability testing), as these will have an impact on adoption/use rates and help with interpreting results.
Does your paper address subitem 5-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study We reviewed physician review websites before generating the stimuli.

5-iii) Revisions and updating
Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was "frozen" during the trial. Describe dynamic components such as news feeds or changing content which may have an impact on the replicability of the intervention (for unexpected events see item 3b). Does your paper address subitem 5-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable. We examined physician rating websites in general, not a speci c one.

5-iv) Quality assurance methods
Provide information on quality assurance methods to ensure accuracy and quality of information provided [1], if applicable.
Does your paper address subitem 5-iv?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study We used attention checks and manipulation checks. All measures were reliable. 5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing owcharts of the algorithms used Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing owcharts of the algorithms used. Replicability (i.e., other researchers should in principle be able to replicate the study) is a hallmark of scienti c reporting.
Does your paper address subitem 5-v?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Screenshots of the stimuli were provided in the manuscript.

5-vi) Digital preservation
Digital preservation: Provide the URL of the application, but as the intervention is likely to change or disappear over the course of the years; also make sure the intervention is archived (Internet Archive, webcitation.org, and/or publishing the source code or screenshots/videos alongside the article). As pages behind login screens cannot be archived, consider creating demo pages which are accessible without login.
Does your paper address subitem 5-vi?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Screenshots were provided.

5-vii) Access
Access: Describe how participants accessed the application, in what setting/context, if they had to pay (or were paid) or not, whether they had to be a member of speci c group. If known, describe how participants obtained "access to the platform and Internet" [1]. To ensure access for editors/reviewers/readers, consider to provide a "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes, see vi).
Does your paper address subitem 5-vii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Participants accessed the stimuli via Qualtrics. Does your paper address subitem 5-viii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Following consent, participants were presented a cover story to read. On the basis of the type of physician that they were assigned to, participants were asked to imagine themselves in a situation looking for either a primary care physician or a surgeon. The vignette about a primary care physician depicted a situation that the participant recently moved to a new city and was in need of a new primary care physician. Owing to a lack of input from family members and friends, they decided to search for primary care physicians on PRWs. The vignette about a surgeon described a situation in which the participant had lasting back pains. The primary care physician suspected that the patient may need spinal surgery and provided a list of surgeons to choose from. The participant decided to search for the recommended surgeons on PRWs. After reading through the scenario and imaging themselves in the described situation, each participant was directed to a physician review page to learn about the physician. A total of 16 physician review pages were developed for this study (see Figure 1).
The top part of each page listed basic information about a physician, including the physician's name (Dr J Smith), the specialty (family medicine or surgeon), and information on new patient acceptance (accepting new patients). To manipulate the type of a physician, half of the Web pages listed the physician's specialty as family medicine and the other half described the physician as a surgeon.
Each page contained 4 aggregated rating categories about Dr Smith, including 2 items on technical skills ("My doctor accurately diagnosed my problem" and "My doctor effectively treated my problem") and 2 on interpersonal skills ("My doctor was caring" and "My doctor spent enough time with me"). Past research has suggested that a physician's skills on diagnosis and treatment are among the most important considerations when selecting a physician [32]. In addition, a physician's personal manner as well as time spent with a patient are critical to a patient's satisfaction on the physician's interpersonal skills [32,33]. These 4 categories frequently appear on PRWs [34] and thus are adopted in this study. To manipulate the valence of physician ratings, these rating categories were assigned different star ratings. Each rating category was presented in the form of aggregated ratings. In the conditions where a physician received high ratings on technical skills, the 2 items pertaining to technical skills were given 5/5-star ratings. In the conditions of moderate ratings on technical skills, the same items were assigned 3/5-star ratings.
We chose to examine moderate ratings instead of low ratings in this study because research suggests that low ratings are relatively uncommon on PRWs . The valence of a physician's interpersonal skills was manipulated in the same way. Furthermore, essential the rating categories were presented to participants in counterbalanced order to control for the impact of rating order effects. In half of the experimental conditions, the 2 rating categories on technical skills were displayed before the 2 categories on interpersonal skills. In the other half, ratings on technical skills were presented beneath the ratings on interpersonal skills. "

5-ix) Describe use parameters
Describe use parameters (e.g., intended "doses" and optimal timing for use). Clarify what instructions or recommendations were given to the user, e.g., regarding timing, frequency, heaviness of use, if any, or was the intervention used ad libitum.
Does your paper address subitem 5-ix?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Participants were instructed to carefully read the stimuli before completing the survey.

5-x) Clarify the level of human involvement
Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21 -generalizability). Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study fully-automated.

5-xi) Report any prompts/reminders used
Report any prompts/reminders used: Clarify if there were prompts (letters, emails, phone calls, SMS) to use the application, what triggered them, frequency etc. It may be necessary to distinguish between the level of prompts/reminders required for the trial, and the level of prompts/reminders for a routine application outside of a RCT setting (discuss under item 21 -generalizability).
Does your paper address subitem 5-xi? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not used.

5-xii) Describe any co-interventions (incl. training/support)
Describe any co-interventions (incl. training/support): Clearly state any interventions that are provided in addition to the targeted eHealth intervention, as ehealth intervention may not be designed as standalone intervention. This includes training sessions and support [1]. It may be necessary to distinguish between the level of training required for the trial, and the level of training for a routine application outside of a RCT setting (discuss under item 21 -generalizability. Does your paper address subitem 5-xii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study subitem not at all important 1 2 3 4 5 essential p y Participants were also asked 2 questions about their past experiences about looking for or having a primary care physician or back surgeon based on the physician type they were assigned to (eg, "Have you ever had a primary care physician or back surgeon?", "Have you ever searched for a primary care physician or back surgeon?"). Participants answered either yes (1) or no (2) to both questions. For participants assigned to conditions involving a primary care physician, 76.6% (209/273) reported that they have had a primary care physician and 76.2% (208/273) reported that they have searched for a primary care physician. For participants assigned to conditions involving a back surgeon, only 5.2% (14/267) reported that they have had a back surgeon and 12.4% (33/267) reported that they have searched for a back surgeon.
Perceived Reliability of Ratings Previous research has suggested that people may perceive the reliability of Webbased ratings differently [3,35], which, in turn, may affect their willingness to choose a physician. To control for the variation, participants were asked 1 question to assess the extent to which they consider the Web-based ratings reliable (ie, "To what extent do you consider the patient ratings are reliable measures of Dr Smith's quality?"). The item was rated on a 7-point scale with the anchors 1=not reliable at all and 7=completely reliable (mean 5.14, SD 1.05). " 6a-ii) Describe whether and how "use" (including intensity of use/dosage) was de ned/measured/monitored Describe whether and how "use" (including intensity of use/dosage) was de ned/measured/monitored (logins, log le analysis, etc.). Use/adoption metrics are important process outcomes that should be reported in any ehealth trial.

Does your paper address subitem 6a-ii?
Copy and paste relevant sections from manuscript text Use means reviewing physician ratings sites. Does your paper address CONSORT subitem 6b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study No changes were made.

7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size
Describe whether and how expected attrition was taken into account when calculating the sample size.
https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrTSkZQL2-3O8O9hrL5Sw/viewform?hl=en_US&formkey=dGlKd2… 34/59 7b) When applicable, explanation of any interim analyses and stopping guidelines 8a) Method used to generate the random allocation sequence NPT: When applicable, how care providers were allocated to each trial group 8b) Type of randomisation; details of any restriction (such as blocking and block size) Does your paper address subitem 7a-i?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Sample size was determined based on the number of experimental conditions. Does your paper address CONSORT subitem 7b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable.
Does your paper address CONSORT subitem 8a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Randomiastion function in Qualtrics was used.
Does your paper address CONSORT subitem 8b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study participants were randomly assigned to one experimental condition. Most survey items were randomized. 9) Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned 10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions 11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 9? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable Does your paper address CONSORT subitem 10? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The survey system Qualtrics did the above tasks.
11a-i) Specify who was blinded, and who wasn't Specify who was blinded, and who wasn't. Usually, in web-based trials it is not possible to blind the participants [1, 3] (this should be clearly acknowledged), but it may be possible to blind outcome assessors, those doing data analysis or those administering co-interventions (if any). subitem not at all important 1 2 3 4 5 essential 11b) If relevant, description of the similarity of interventions (this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention) Does your paper address subitem 11a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study participants 11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator" Informed consent procedures (4a-ii) can create biases and certain expectations -discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator".
Does your paper address subitem 11a-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study No, they didn't know the differences.
Does your paper address CONSORT subitem 11b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study not applicable 12a-i) Imputation techniques to deal with attrition / missing values Imputation techniques to deal with attrition / missing values: Not all participants will use the intervention/comparator as intended and attrition is typically high in ehealth trials. Specify how participants who did not use the application or dropped out from the trial were treated in the statistical analysis (a complete case analysis is strongly discouraged, and simple imputation techniques such as LOCF may also be problematic [4]).
Does your paper address subitem 12a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable

RESULTS
13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center Does your paper address subitem X26-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Consent was obtained online.

X26-iii) Safety and security procedures
Safety and security procedures, incl. privacy considerations, and any steps taken to reduce the likelihood or detection of harm (e.g., education and training, availability of a hotline) Does your paper address subitem X26-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Data were collected anonymously.
Does your paper address CONSORT subitem 13a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The information was provided in Table 1. Preferably, this is shown in a CONSORT ow diagram) * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "A total of 608 participants completed the Web-based experiment. Participants were recruited via the crowdsourcing website, Amazon's Mechanical Turk (mTurk), and compensated for their time. We excluded people who failed the attention checks (n=26) and those who spent no time (n=1) or less than 5 seconds on the Web page (n=41). "

13b-i) Attrition diagram
Strongly recommended: An attrition diagram (e.g., proportion of participants still logging in or using the intervention/comparator in each group plotted over time, similar to a survival curve) or other gures or tables demonstrating usage/dose/engagement. Does your paper address subitem 13b-i?
Copy and paste relevant sections from the manuscript or cite the gure number if applicable (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 14a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study March -May 2017 14a-i) Indicate if critical "secular events" fell into the study period Indicate if critical "secular events" fell into the study period, e.g., signi cant changes in Internet resources available or "changes in computer hardware or Internet delivery resources" Does your paper address subitem 14a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 14b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Does your paper address CONSORT subitem 15? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The information was provided in Table 1.

15-i) Report demographics associated with digital divide issues
In ehealth trials it is particularly important to report demographics associated with digital divide issues, such as age, education, gender, social-economic status, computer/Internet/ehealth literacy of the participants, if known.
Does your paper address subitem 15-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The information was provided in Table 1. 16-i) Report multiple "denominators" and provide de nitions Report multiple "denominators" and provide de nitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at speci c pre-de ned time points of interest (in absolute and relative numbers per group). Always clearly de ne "use" of the intervention. subitem not at all important 1 2 3 4 5 essential 17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% con dence interval) Does your paper address subitem 16-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The denominators were reported in the manuscript.

16-ii) Primary analysis should be intent-to-treat
Primary analysis should be intent-to-treat, secondary analyses could include comparing only "users", with the appropriate caveats that this is no longer a randomized sample (see 18-i).
Does your paper address subitem 16-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study subitem not at all important 1 2 3 4 5 essential 17b) For binary outcomes, presentation of both absolute and relative effect sizes is recommended 18) Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-speci ed from exploratory 17a-i) Presentation of process outcomes such as metrics of use and intensity of use In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational de nitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "average session length". These must be accompanied by a technical description how a metric like a "session" is de ned (e.g., timeout after idle time) [1] (report under item 6a).
Does your paper address subitem 17a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 17b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable.
Does your paper address CONSORT subitem 18? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Data were provided in the results section.

18-i) Subgroup analysis of comparing only users
A subgroup analysis of comparing only users is not uncommon in ehealth trials, but if done, it must be stressed that this is a self-selected sample and no longer an unbiased sample from a randomized trial (see 16-iii).
Does your paper address subitem 18-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 19? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study No important harms or unintended effects in each group.

19-i) Include privacy breaches, technical problems
Include privacy breaches, technical problems. This does not only include physical "harm" to participants, but also incidents such as perceived or real privacy breaches [1], technical problems, and other unexpected/unintended incidents. "Unintended effects" also includes unintended positive effects [2]. Does your paper address subitem 22-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Your answer

20-i) Typical limitations in ehealth trials
Typical limitations in ehealth trials: Participants in ehealth trials are rarely blinded. Ehealth trials often look at a multiplicity of outcomes, increasing risk for a Type I error. Discuss biases due to non-use of the intervention/usability issues, biases through informed consent procedures, unexpected events. PRWs, many portals also allow patients to leave narrative comments to detail their satisfaction and dissatisfaction. Aggregated numerical ratings tend to provide patients a holistic view of physicians and the services they provide. Narrative comments, on the contrary, can capture more detailed and nuanced feedback that is not re ected in structured rating systems. A direction for future research is to investigate how numerical ratings and narrative comments work together to affect people's willingness to choose a physician, especially if 2 sources present contradictory information.
Third, this study focused on rating categories pertaining to a physician's technical and interpersonal skills. In selection of a physician, patients take into account many considerations beyond a physician's quali cations. For example, previous research has found that management practices such as punctuality and staff quality are also considered in patients' choice of physicians [31,39]. Besides reviewing a physician's quali cations, many PRWs also include rating categories on management practices.
Future research should look into these aspects in addition to a physician's skills.
Fourth, despite a wide range of specialties, this study operationalized a specialist to be a back surgeon. However, it is likely that patients use different selection criteria Finally, a patient's willingness to choose a physician is in uenced by a variety of factors beyond numerical ratings displayed on PRWs. For example, demographic information of a physician (eg, sex and age) and environmental factors (eg, o ce location) should be taken into account when examining patients' choice of physicians. Another direction for future research is to explore underlying mechanisms, especially perceptual processes, through which physician types and patient reviews affect people's choice of physicians. "

21-i) Generalizability to other populations
Generalizability to other populations: In particular, discuss generalizability to a general Internet population, outside of a RCT setting, and general patient population, including applicability of the study results for other organizations Does your paper address subitem 21-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Your answer Discuss if there were elements in the RCT that would be different in a routine application setting (e.g., prompts/reminders, more human involvement, training sessions or other co-interventions) and what impact the omission of these elements could have on use, adoption, or outcomes if the intervention is applied outside of a RCT setting.
Does your paper address subitem 21-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 23? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study ISRCTN91316463 "The impact of web-based ratings on physician selection" Does your paper address CONSORT subitem 24? * Cite a Multimedia Appendix, other reference, or copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study It can be accessed upon request to the researchers. About the CONSORT EHEALTH checklist Does your paper address CONSORT subitem 25? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable.
X27-i) State the relation of the study team towards the system being evaluated In addition to the usual declaration of interests ( nancial or otherwise), also state the relation of the study team towards the system being evaluated, i.e., state if the authors/evaluators are distinct from or identical with the developers/sponsors of the intervention.
Does your paper address subitem X27-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study STOP -Save this form as PDF before you click submit To generate a record that you lled in this form, we recommend to generate a PDF of this page (on a Mac, simply select "print" and then select "print as PDF") before you submit it.
When you submit your (revised) paper to JMIR, please upload the PDF as supplementary le.
Don't worry if some text in the textboxes is cut off, as we still have the complete information in our database. Thank you!