A Web-Based Patient Portal for Mental Health Care: Benefits Evaluation

Background Treatment for mental illness has shifted from focusing purely on treatment of symptoms to focusing on personal recovery. Patient activation is an important component of the recovery journey. Patient portals have shown promise to increase activation in primary and acute care settings, but the benefits to tertiary level mental health care remain unknown. Objective To conduct a benefits evaluation of a Web-based portal for patients undergoing treatment for serious or persistent mental illness in order to examine the effects on (1) patient activation, (2) recovery, (3) productivity, and (4) administrative efficiencies. Methods All registered inpatients and outpatients at a tertiary level mental health care facility were offered the opportunity to enroll and utilize the patient portal. Those who chose to use the portal and those who did not were designated as “users” and “nonusers,” respectively. All patients received usual treatment. Users had Web-based access to view parts of their electronic medical record, view upcoming appointments, and communicate with their health care provider. Users could attend portal training or support sessions led by either the engagement coordinator or peer support specialists. A subset of patients who created and utilized their portal account completed 2 Web-based surveys at baseline (just after enrollment; n=91) and at follow-up (6 and 10 months; n=65). The total score of the Mental Health Recovery Measure (MHRM) was a proxy for patient activation and the individual domains measured recovery. The System and Use Survey Tool (SUS) examined the use of functions and general feedback about the portal. Organizational efficiencies were evaluated by examining the odds of portal users and nonusers missing appointments (productivity) or requesting information from health information management (administrative efficiencies) in the year before (2014) and the year after (2015) portal implementation. Results A total of 461 patients (44.0% male, n=203) registered for the portal, which was used 4761 times over the 1-year follow-up period. The majority of uses (95.34%, 4539/4761) were for e-views. The overall MHRM score increased from 70.4 (SD 23.6) at baseline to 81.7 (SD 25.1) at combined follow-up (P=.01). Of the 8 recovery domains, 7 were increased at follow-up (all P<.05). The odds of a portal user attending an appointment were 67% (CI 56%-79%) greater than that of nonusers over the follow-up period. Compared with 2014, over 2015 there was an 86% and 57% decrease in requests for information in users and nonusers, respectively. The SUS revealed that users felt an increased sense of autonomy and found the portal to be user-friendly, helpful, and efficient but felt that more information should be accessible. Conclusions The benefits evaluation suggested that access to personal health records via patient portals may improve patient activation, recovery scores, and organizational efficiencies in a tertiary level mental health care facility.

TITLE AND ABSTRACT 1a) TITLE: Identi멹cation as a randomized trial in the title 1a) Does your paper address CONSORT item 1a? * I.e does the title contain the phrase "Randomized Controlled Trial"? (if not, explain the reason under "other") yes Other: 1a-i) Identify the mode of delivery in the title Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internet-based" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if ofᝳ昃ine products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms. 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1a-i? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "A webbased patient portal "

1a-ii) Non-web-based components or important co-interventions in title
Mention non-web-based components or important co-interventions in title, if any (e.g., "with telephone support").
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1a-ii?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study N/A the portal is a nonmandatory supplement to treatment. There is no human or other support specifically associated with portal use.
1a-iii) Primary condition or target group in the title Mention primary condition or target group in the title, if any (e.g., "for children with Type I Diabetes") Example: A Web-based and Mobile Intervention with Telephone Support for Children with Type I Diabetes: Randomized Controlled Trial 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1a-iii? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "mental health care" 1b) ABSTRACT: Structured summary of trial design, methods, results, and conclusions NPT extension: Description of experimental treatment, comparator, care providers, centers, and blinding status.

1b-i) Key features/functionalities/components of the intervention and comparator in the METHODS section of the ABSTRACT
Mention key features/functionalities/components of the intervention and comparator in the abstract. If possible, also mention theories and principles used for designing the site. Keep in mind the needs of systematic reviewers and indexers by including important synonyms. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1b-i? * Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Those who chose to use the portal or not were designated as "users" and "nonusers", respectively. All patients received usual treatment. Users additionally had webbased access to the portal" 1b-ii) Level of human involvement in the METHODS section of the ABSTRACT Clarify the level of human involvement in the abstract, e.g., use phrases like "fully automated" vs. "therapist/nurse/care provider/physician-assisted" (mention number and expertise of providers involved, if any). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1b-ii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Users could attend portal training/support sessions led by either the Engagement Coordinator or Peer Support Specialists" 1b-iii) Open vs. closed, web-based (self-assessment) vs. face-to-face assessments in the METHODS section of the ABSTRACT Mention how participants were recruited (online vs. ofᝳ昃ine), e.g., from an open access website or from a clinic or a closed online user group (closed usergroup trial), and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment). Clearly say if outcomes were self-assessed through questionnaires (as common in web-based trials). Note: In traditional ofᝳ昃ine trials, an open trial (open-label trial) is a type of clinical trial in which both the researchers and participants know which treatment is being administered. To avoid confusion, use "blinded" or "unblinded" to indicated the level of blinding instead of "open", as "open" in web-based trials usually refers to "open access" (i.e. participants can self-enrol). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1b-iii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study https://docs.google.com/forms/d/e/1FAIpQLSfZBSUp1bwOc_OimqcS64RdfIAFvmrTSkZQL23O8O9hrL5Sw/viewform?hl=en_US&formkey=dGlKd2Z2Q1lN… 6/38 "...completed two webbased surveys at baseline (just after enrollment) and at followup..." 1b-iv) RESULTS section in abstract must contain use data Report number of participants enrolled/assessed in each group, the use/uptake of the intervention (e.g., attrition/adherence metrics, use over time, number of logins etc.), in addition to primary/secondary outcomes. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1b-iv?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "461 patients (44% male) registered for the portal, which was used 4761 times over the oneyear followup period" 1b-v) CONCLUSIONS/DISCUSSION in abstract for negative trials Conclusions/Discussions in abstract for negative trials: Discuss the primary outcome -if the trial is negative (primary outcome not changed), and the intervention was not used, discuss whether negative results are attributable to lack of uptake and discuss reasons. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) 3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons Does your paper address CONSORT subitem 3b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study N/A there were no changed to the methods after trial commencement.
3b-i) Bug 멹xes, Downtimes, Content Changes Bug 멹xes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug 멹xes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have inᝳ昃uenced study design such as staff changes, system failures/downtimes, etc. [2].
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 3b-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study ". Version 1 of the portal was released to the organization on December, 2014. Nine users pilot tested the portal in December 2014 and provided feedback before it was rolled out to the entire organization later that month. Increased functionality and updates will be available with the next scheduled upgrade in November 1st, 2016. There were no changes to the portal over the course of the data collection period (Meditech 6.07, Portal 1.0). "

4a) Eligibility criteria for participants
Does your paper address CONSORT subitem 4a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "All inpatients & outpatients (or their carers) receiving care from December 2014December 2015 were eligible to register for a portal account and were invited to participate."

4a-i) Computer / Internet literacy
Computer / Internet literacy is often an implicit "de facto" eligibility criterion -this should be explicitly clari멹ed.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 4a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study N/A this was not an eligibility criterion, though it may have influenced who chose to sign up or not.
4a-ii) Open vs. closed, web-based vs. face-to-face assessments: Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. ofᝳ昃ine), e.g., from an open access website or from a clinic, and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasianonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email con멹rmation, phone calls) were used to detect/prevent these.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 4a-ii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Patients were approached by the Engagement Coordinator (Clinical Educator) and provided with information about the portal." "Demographics for the overall populations were extracted from the EMR data repository via structured query language (SQL) report. Demographics for the subset of individuals completing the webbased surveys was self reported via survey." 4a-iii) Information giving during recruitment Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 4a-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study No.
4b) Settings and locations where the data were collected Does your paper address CONSORT subitem 4b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "was carried out in a tertiary level mental health care facility (Ontario Shores Centre for Mental Health Sciences, Whitby, Canada), " 4b-i) Report if outcomes were (self-)assessed through online questionnaires Clearly report if outcomes were (self-)assessed through online questionnaires (as common in web-based trials) or otherwise.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 4b-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Portal users were prompted to complete the Mental Health Recovery Measure (MHRM©) at registration and six and ten months following portal registration. A link to the webbased survey (http://www.surveymonkey.com; SurveyMonkey Inc., Pal Alto, USA) was available on the portal. " "A link to the System & Use Survey (SUS) was available on the portal six and ten months after portal registration to examine users' experiences with evisits, eviews and erequest for prescription refills. " 4b-ii) Report how institutional af멹liations are displayed Does your paper address subitem 4b-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not reported, but only Ontario Shores brands were included and only Ontario Shores' patients were eligible.

5)
The interventions for each group with suf멹cient details to allow replication, including how and when they were actually administered 5-i) Mention names, credential, af멹liations of the developers, sponsors, and owners Mention names, credential, af멹liations of the developers, sponsors, and owners [6] (if authors/evaluators are owners or developer of the software, this needs to be declared in a "Conᝳ昃ict of interest" section or mentioned elsewhere in the manuscript).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "The portal implementation project was sponsored by Canada Health Infoway (CHI) and developed by Meditech "

5-ii) Describe the history/development process
Describe the history/development process of the application and previous formative evaluations (e.g., focus groups, usability testing), as these will have an impact on adoption/use rates and help with interpreting results.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Version 1 of the portal was released to the organization on December, 2014. Nine users pilot tested the portal in December 2014 and provided feedback before it was rolled out to the entire organization later that month. Increased functionality and updates will be available with the next scheduled upgrade in November 1st, 2016. There were no changes to the portal over the course of the data collection period (Meditech 6.07, Portal 1.0). "

5-iii) Revisions and updating
Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was "frozen" during the trial.
Describe dynamic components such as news feeds or changing content which may have an impact on the replicability of the intervention (for unexpected events see item 3b).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "(Meditech 6.07, Portal 1.0). "

5-iv) Quality assurance methods
Provide information on quality assurance methods to ensure accuracy and quality of information provided [1], if applicable.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-iv?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not reported 5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screencapture video, and/or providing ᝳ昃owcharts of the algorithms used Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing ᝳ昃owcharts of the algorithms used. Replicability (i.e., other researchers should in principle be able to replicate the study) is a hallmark of scienti멹c reporting.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-v?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not reported.

5-vi) Digital preservation
Digital preservation: Provide the URL of the application, but as the intervention is likely to change or disappear over the course of the years; also make sure the intervention is archived (Internet Archive, webcitation.org, and/or publishing the source code or screenshots/videos alongside the article). As pages behind login screens cannot be archived, consider creating demo pages which are accessible without login.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-vi?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not reported

5-vii) Access
Access: Describe how participants accessed the application, in what setting/context, if they had to pay (or were paid) or not, whether they had to be a member of speci멹c group. If known, describe how participants obtained "access to the platform and Internet" [1]. To ensure access for editors/reviewers/readers, consider to provide a "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes, see vi).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-vii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Enrollment was limited to inpatients or outpatients registered at Ontario Shores and their proxy users. Access and services were provided at no cost to the patient, and users continued to have access to their records following discharge." subitem not at all important essential Does your paper address subitem 5-viii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Users accessed the portal online at https://healthcheck.ontarioshores.ca through any internet browser on a computer or electronic device with internet connectivity. Users were able to show, print and share their record with health care providers at other facilities in support of maintaining continuity of care. For the purpose of this evaluation, three functional components were defined: 1) Eviews: Allowed users to view parts of their electronic health record, including reports, discharge summaries, allergies, demographics, and their ambulatory medication list. They could also 5-ix) Describe use parameters Describe use parameters (e.g., intended "doses" and optimal timing for use). Clarify what instructions or recommendations were given to the user, e.g., regarding timing, frequency, heaviness of use, if any, or was the intervention used ad libitum. not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Recommended frequency of use was not specified to users; rather, it was recommended to utilize educational resources, pamphlets, website and other materials to support them with navigation within the system, as needed. Users received email notifications when new information was available in the portal."

5-x) Clarify the level of human involvement
Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21 -generalizability).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-x?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study training through a video and access to a demonstration account to experience portal use and a Learning Management System module. Clinicians received further training through the Medical Advisory Council, Nursing Council, other professional councils and onunit services. Training was included in clinical orientation for new hires. A process to support clinicians with enrolling users was built into the EMR. Formal training was not provided to users, but support sessions were facilitated by Peer Support Specialists and the Engagement Coordinator and available for users to attend on an ongoing basis. "

5-xi) Report any prompts/reminders used
Report any prompts/reminders used: Clarify if there were prompts (letters, emails, phone calls, SMS) to use the application, what triggered them, frequency etc. It may be necessary to distinguish between the level of prompts/reminders required for the trial, and the level of prompts/reminders for a routine application outside of a RCT setting (discuss under item 21 -generalizability).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-xi? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Users received email notifications when new information was available in the portal." 5-xii) Describe any co-interventions (incl. training/support) Describe any co-interventions (incl. training/support): Clearly state any interventions that are provided in addition to the targeted eHealth intervention, as ehealth intervention may not be designed as stand-alone intervention. This includes training sessions and support [1]. It may be necessary to distinguish between the level of training required for the trial, and the level of training for a routine application outside of a RCT setting (discuss under item 21 -generalizability.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-xii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study " Formal training was not provided to users, but support sessions were facilitated by Peer Support Specialists and the Engagement Coordinator and available for users to attend on an ongoing basis. " 6a) Completely de멹ned pre-speci멹ed primary and secondary outcome measures, including how and when they were assessed Does your paper address CONSORT subitem 6a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Demographics: Demographics for the overall populations were extracted from the EMR data repository via structured query language (SQL) report. Demographics for the subset of individuals completing the webbased surveys was selfreported via survey.
Usage: The number of patients who registered for the portal and number of times each function was used was pulled from the data repository via SQL reports.
6a-i) Online questionnaires: describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed If outcomes were obtained through online questionnaires, describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed [9]. 6a-iii) Describe whether, how, and when qualitative feedback from participants was obtained Describe whether, how, and when qualitative feedback from participants was obtained (e.g., through emails, feedback forms, interviews, focus groups).

2 3 4 5
subitem not at all important essential Does your paper address subitem 6a-iii?
Copy and paste relevant sections from manuscript text "Qualitative Portal Experience: Free text answers to the SUS (administered as described above) provided qualitative feedback regarding experiences with portal use." above described that it was webbased survey 6b) Any changes to trial outcomes after the trial commenced, with reasons Does your paper address CONSORT subitem 6b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study There were no changes.
7a) How sample size was determined NPT: When applicable, details of whether and how the clustering by care provides or centers was addressed 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size Describe whether and how expected attrition was taken into account when calculating the sample size.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 7a-i?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "The entire organizational patient population was used for observation. The target sample size for survey completion was 60, based on the CHI (study sponsor) statement of work." 7b) When applicable, explanation of any interim analyses and stopping guidelines Does your paper address CONSORT subitem 7b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable 8a) Method used to generate the random allocation sequence NPT: When applicable, how care providers were allocated to each trial group Does your paper address CONSORT subitem 8a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable 8b) Type of randomisation; details of any restriction (such as blocking and block size) Does your paper address CONSORT subitem 8b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable 9) Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned Does your paper address CONSORT subitem 9? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable 10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions Does your paper address CONSORT subitem 10? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable 11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment 11a-i) Specify who was blinded, and who wasn't Specify who was blinded, and who wasn't. Usually, in web-based trials it is not possible to blind the participants [1, 3] (this should be clearly acknowledged), but it may be possible to blind outcome assessors, those doing data analysis or those administering co-interventions (if any). Does your paper address subitem 11a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study

No blinding
11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator" Informed consent procedures (4a-ii) can create biases and certain expectations -discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator".
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 11a-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not reported 11b) If relevant, description of the similarity of interventions (this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention) Does your paper address CONSORT subitem 11b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study 12a) Statistical methods used to compare groups for primary and secondary outcomes NPT: When applicable, details of whether and how the clustering by care providers or centers was addressed Does your paper address CONSORT subitem 12a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study inputted to OpenEpi (Version 3) [18] to calculate the odds ratio (OR) for the 2014 and 2015 data.
Ttests were used to examine changes in the overall MHRM© and each of the recovery domains recovery domains [19]. Basic coding was completed for the freetext sections of the SUS. As participation in the 10 months or more followup surveys was low, responses to 6 and 10 months or more followup surveys were combined for MHRM© and SUS analyses to meet CHI requirements." 12a-i) Imputation techniques to deal with attrition / missing values Imputation techniques to deal with attrition / missing values: Not all participants will use the intervention/comparator as intended and attrition is typically high in ehealth trials. Specify how participants who did not use the application or dropped out from the trial were treated in the statistical analysis (a complete case analysis is strongly discouraged, and simple imputation techniques such as LOCF may also be problematic [4]).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 12a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable. Due to study design, those who chose not to use were designated as nonusers.

12b) Methods for additional analyses, such as subgroup analyses and adjusted analyses
Does your paper address CONSORT subitem 12b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Informed consent was waived by the research ethics board for analysis of deidentified data pulled from the organization's EMR. Consent to participate in a subset of participants completing webbased surveys was implied by completing the surveys after being invited by detailed email communication from the study coordinator."

X26-iii) Safety and security procedures
Safety and security procedures, incl. privacy considerations, and any steps taken to reduce the likelihood or detection of harm (e.g., education and training, availability of a hotline) 1 2 3 4 5 subitem not at all important essential Does your paper address subitem X26-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study

RESULTS
13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center Does your paper address CONSORT subitem 13a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study

Yes (See tables 2 & 3)
13b) For each group, losses and exclusions after randomisation, together with reasons Does your paper address CONSORT subitem 13b? (NOTE: Preferably, this is shown in a CONSORT ᝳ昃ow diagram) * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable

13b-i) Attrition diagram
Strongly recommended: An attrition diagram (e.g., proportion of participants still logging in or using the intervention/comparator in each group plotted over time, similar to a survival curve) or other 멹gures or tables demonstrating usage/dose/engagement.

2 3 4 5
subitem not at all important essential Does your paper address subitem 13b-i?
Copy and paste relevant sections from the manuscript or cite the 멹gure number if applicable (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable 14a) Dates de멹ning the periods of recruitment and followup Does your paper address CONSORT subitem 14a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study 14a-i) Indicate if critical "secular events" fell into the study period Indicate if critical "secular events" fell into the study period, e.g., signi멹cant changes in Internet resources available or "changes in computer hardware or Internet delivery resources" 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 14a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "There were no changes to the portal over the course of the data collection period (Meditech 6.07, Portal 1.0). " 14b) Why the trial ended or was stopped (early) Does your paper address CONSORT subitem 14b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Table 2.

15-i) Report demographics associated with digital divide issues
In ehealth trials it is particularly important to report demographics associated with digital divide issues, such as age, education, gender, social-economic status, computer/Internet/ehealth literacy of the participants, if known.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 15-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Only sex and age were collected and are reported in Table 2. 16) For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups 16-i) Report multiple "denominators" and provide de멹nitions Report multiple "denominators" and provide de멹nitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at speci멹c pre-de멹ned time points of interest (in absolute and relative numbers per group). Always clearly de멹ne "use" of the intervention.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 16-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Table 3 16-ii) Primary analysis should be intent-to-treat Primary analysis should be intent-to-treat, secondary analyses could include comparing only "users", with the appropriate caveats that this is no longer a randomized sample (see 18-i).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 16-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Cohort not applicable.
17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% con멹dence interval) Does your paper address CONSORT subitem 17a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "In 2014 (the year prior to the portal launch), the odds of a user attending a scheduled appointment were 17% greater than nonusers (OR: 1.17, 95%CI 1.08, 1.26). In 2015 (the year of the followup period), the odds of a user attending a scheduled appointment were 67% greater than nonusers (OR: 1.67; 95%CI 1.56, 1.79)." 17a-i) Presentation of process outcomes such as metrics of use and intensity of use In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational de멹nitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "average session length". These must be accompanied by a technical description how a metric like a "session" is de멹ned (e.g., timeout after idle time) [1] (report under item 6a).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 17a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Over the yearlong followup period, 461 service users (44% male; n=203) registered for the portal and were designated as users. The majority of users were between the ages of 25 and 34 years. The portal was used 4761 times with the majority of logins for eviews (n=4539; 95.3%), followed by evisits (n=210; 4.4%) and erenewal of prescriptions (n=12; 0.3%)." 17b) For binary outcomes, presentation of both absolute and relative effect sizes is recommended Does your paper address CONSORT subitem 17b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable.
18) Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-speci멹ed from exploratory Does your paper address CONSORT subitem 18? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study

18-i) Subgroup analysis of comparing only users
A subgroup analysis of comparing only users is not uncommon in ehealth trials, but if done, it must be stressed that this is a self-selected sample and no longer an unbiased sample from a randomized trial (see 16-iii).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 18-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "The design of this study may introduce bias when comparing portal users versus nonportal users for organizational measures as the users were interested in and motivated to use the portal, which may translate into increased interest and motivation to participate in treatment. "

19) All important harms or unintended effects in each group
(for speci멹c guidance see CONSORT for harms) Does your paper address CONSORT subitem 19? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Table 4 DISCUSSION 22) Interpretation consistent with results, balancing bene멹ts and harms, and considering other relevant evidence NPT: In addition, take into account the choice of the comparator, lack of or partial blinding, and unequal expertise of care providers or centers in each group 22-i) Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use) Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use). Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study an EMR linked portal for inpatients and outpatients receiving services at a tertiary facility specializing in severe and persistent mental illness.
The novel findings of this study are that implementation of the portal for inpatients and outpatients resulted in activation of service users and/or carers and in improved recovery scores according to the MHRM© domains. At the organizational level, productivity was increased with fewer missed appointments, and administrative efficiencies were realized with a reduced number of requests of information in the year following compared to the year prior to portal implementation."

22-ii) Highlight unanswered new questions, suggest future research
Highlight unanswered new questions, suggest future research.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 22-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "The fact, however, that this study elicited similar changes in MHRM© as an intensive wellness management and recovery program, suggests this is an important topic for future research." "Future research is planned to conduct focus groups to more thoroughly examine patient experiences and to examine longitudinal effects of increased portal functionalities on mental health symptoms, recovery and healthcare utilization." 20) Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses

20-i) Typical limitations in ehealth trials
Typical limitations in ehealth trials: Participants in ehealth trials are rarely blinded. Ehealth trials often look at a multiplicity of outcomes, increasing risk for a Type I error. Discuss biases due to non-use of the intervention/usability issues, biases through informed consent procedures, unexpected events.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 20-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study surrogate measure since budgetary constraints prevented use of the PAM. Convenience sampling was used to recruit the subset of users completing the SUS and this subset was not necessarily representative of all the users. Additionally, the administration of the surveys via anonymous online survey software ensured confidentiality, but prevented analysis using repeated measures design. It is unknown how many (if any) of the users completed the survey both at baseline and followup or if these samples are different in composition. Demographics suggest compositions were similar. " 21) Generalisability (external validity, applicability) of the trial 멹ndings NPT: External validity of the trial 멹ndings according to the intervention, comparators, patients, and care providers or centers involved in the trial

21-i) Generalizability to other populations
Generalizability to other populations: In particular, discuss generalizability to a general Internet population, outside of a RCT setting, and general patient population, including applicability of the study results for other organizations 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 21-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study "Since the entire patient population of the hospital and its associated clinics were followed for the duration of this study, it has high internal validity. Results may be generalizable to other tertiary care mental health hospitals and outpatient clinics with similar organizational context; however, since the results are specific to this organization, generalizability to other contexts may be limited. "

21-ii) Discuss if there were elements in the RCT that would be different in a routine application setting
Discuss if there were elements in the RCT that would be different in a routine application setting (e.g., prompts/reminders, more human involvement, training sessions or other co-interventions) and what impact the omission of these elements could have on use, adoption, or outcomes if the intervention is applied outside of a RCT setting.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 21-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study Not applicable 24) Where the full trial protocol can be accessed, if available Does your paper address CONSORT subitem 24? * Cite a Multimedia Appendix, other reference, or copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study not available 25) Sources of funding and other support (such as supply of drugs), role of funders Does your paper address CONSORT subitem 25? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study This project was sponsored by Canada Health Infoway, who provided financial support and the framework and guidance for the benefits evaluation.

X27) Conᝳ昃icts of Interest (not a CONSORT item)
X27-i) State the relation of the study team towards the system being evaluated In addition to the usual declaration of interests (멹nancial or otherwise), also state the relation of the study team towards the system being evaluated, i.e., state if the authors/evaluators are distinct from or identical with the developers/sponsors of the intervention.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem X27-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieᝳ昃y explain why the item is not applicable/relevant for your study None to declare About the CONSORT EHEALTH checklist As a result of using this checklist, did you make changes in your manuscript? * yes, major changes yes, minor changes no What were the most important changes you made as a result of using this checklist?
I had omitted some important information from the abstract. How much time did you spend on going through the checklist INCLUDING making changes in your manuscript *

1.25hrs
As a result of using this checklist, do you think your manuscript has improved? * yes no Other: Would you like to become involved in the CONSORT EHEALTH group? This would involve for example becoming involved in participating in a workshop and writing an "Explanation and Elaboration" document yes no Other: Any other comments or questions on CONSORT EHEALTH I am interested in becoming involved in the CONSORT EHEALTH group, but am unsure if I have the appropriate experience, knowledge and skill. Very happy to learn more about the opportunity.
STOP -Save this form as PDF before you click submit To generate a record that you 멹lled in this form, we recommend to generate a PDF of this page (on a Mac, simply select "print" and then select "print as PDF") before you submit it.
When you submit your (revised) paper to JMIR, please upload the PDF as supplementary 멹le.
Don't worry if some text in the textboxes is cut off, as we still have the complete information in our database. Thank you!