Background: Patient-clinician secure messaging is an important function in patient portals and enables patients and clinicians to communicate on a wide spectrum of issues in a timely manner. With its growing adoption and patient engagement, it is time to comprehensively study the secure messages and user behaviors in order to improve patient-centered care.
Objective: The aim of this paper was to analyze the secure messages sent by patients and clinicians in a large multispecialty health system at Mayo Clinic, Rochester.
Methods: We performed message-based, sender-based, and thread-based analyses of more than 5 million secure messages between 2010 and 2017. We summarized the message volumes, patient and clinician population sizes, message counts per patient or clinician, as well as the trends of message volumes and user counts over the years. In addition, we calculated the time distribution of clinician-sent messages to understand their workloads at different times of a day. We also analyzed the time delay in clinician responses to patient messages to assess their communication efficiency and the back-and-forth rounds to estimate the communication complexity.
Results: During 2010-2017, the patient portal at Mayo Clinic, Rochester experienced a significant growth in terms of the count of patient users and the total number of secure messages sent by patients and clinicians. Three clinician categories, namely “physician—primary care,” “registered nurse—specialty,” and “physician—specialty,” bore the majority of message volume increase. The patient portal also demonstrated growing trends in message counts per patient and clinician. The “nurse practitioner or physician assistant—primary care” and “physician—primary care” categories had the heaviest per-clinician workload each year. Most messages by the clinicians were sent from 7 AM to 5 PM during a day. Yet, between 5 PM and 7 PM, the physicians sent 7.0% (95,785/1,377,006) of their daily messages, and the nurse practitioner or physician assistant sent 5.4% (22,121/408,526) of their daily messages. The clinicians replied to 72.2% (1,272,069/1,761,739) patient messages within 1 day and 90.6% (1,595,702/1,761,739) within 3 days. In 95.1% (1,499,316/1,576,205) of the message threads, the patients communicated with their clinicians back and forth for no more than 4 rounds.
Conclusions: Our study found steady increases in patient adoption of the secure messaging system and the average workload per clinician over 8 years. However, most clinicians responded timely to meet the patients’ needs. Our study also revealed differential patient-clinician communication patterns across different practice roles and care settings. These findings suggest opportunities for care teams to optimize messaging tasks and to balance the workload for optimal efficiency.
A patient portal is a secure online platform that allows patients to conveniently access and manage personal health information and communicate with their clinicians . After the Health Information Technology for Economic and Clinical Health Act of 2009, patient portals have gained widespread adoption by health care systems in the United States [ , ]. In 2017, over 90% of health care organizations including the Veterans Administration, Mass General Brigham, Kaiser Permanente, and Mayo Clinic offered patient portal access to their patients [ ]. Patient portals give patients 24-7 access to their health information (eg, clinical visits, lab test results, medications, and discharge summaries) from anywhere with internet connection [ ] and have been shown to improve patient self-management by promoting the awareness of disease knowledge, status, and progress [ ].
A significant function in patient portals is patient-clinician secure messaging, which enables patients and clinicians to timely communicate on a wide spectrum of issues. Patients use secure messaging to request medical appointments and refill prescriptions online [, ]. Clinicians send patients appointment reminders and promote timely preventative care [ , ]. Patients and clinicians can communicate back and forth on complex situations such as new symptoms, disease follow-ups, medication concerns, and other medical questions. Evidence suggests that secure messaging improves health care efficiency, productivity, and quality. For instance, Zhou et al [ ] investigated more than 4000 users at Kaiser Permanente before and after the introduction of a secure message system and found that their annual rates of in-person primary care visits were reduced by 9.7% after their adoption of secure messaging. Simon et al [ ] showed that both antidepressant adherence rate and depression treatment satisfaction increased by 20% among patients using secure messaging.
On the other hand, the secure messaging resulted in additional workload for clinicians and contributed to work burnout, as it increased patient-clinician interactions between in-person patient visits. According to a survey, 63% of 43 clinicians across 5 clinics disagreed with the notion that “secure messaging reduces my workload,” and 33% agreed with the notion that “secure messaging has a negative effect on my workflow” . Another study shows that primary care physicians spend, on average, 1.4 hours of their workday (5.9 hours) interacting with electronic health records for non–face-to-face care after clinic hours [ ].
With the increase in the number of patients signing up for these portals, the number of secure messages has risen substantially, especially during the COVID-19 pandemic [- ]. It will be critical to the care teams to understand the patient-clinician messaging and to properly distribute the communication load for better efficiency and avoiding clinician burnout. It could be foreseen that some health care systems would be likely to face the challenge of managing the increasing volume of patient messages soon [ ], which will require new billing models and practice metrics, or additional infrastructures, including support staffs to reply to the increasing volume of patient messages. However, there is limited understanding of the use of secure messaging and the extent of users’ interaction through this medium including clinician messaging load, messaging time delay, messaging time distribution in a day, and messaging complexity of a communication thread.
In this study, we attempted to bridge this knowledge gap by analyzing more than 5 million secure messages that were generated by patients and clinicians between 2010 and 2017 at a large multispecialty health system at Mayo Clinic, Rochester. We performed message-oriented and sender-oriented analyses by calculating the message volumes, patient or clinician population sizes, message counts per patient or clinician, time distribution of clinician messaging, and their trends over the years. We also performed thread-oriented analysis to probe the time delay in clinician responses to patient messages to assess their communication efficiency and the back-and-forth rounds to estimate the communication complexity. Our findings shed light on the patient-clinician digital communication and inform future improvement in the use of secure medical messaging.
Data Collection and Preprocessing
The patient portal (Patient Online Services) at Mayo Clinic, Rochester  was started in 2010 for primary care practice and later extended to specialty practice in 2013. The patient portal allows patients and clinicians to communicate bidirectionally via secure messaging on a wide range of issues. We retrieved more than 5 million secure messages from the patient portal between February 18, 2010, and December 31, 2017. Each message has a unique identifier (ID), previous message ID, initial message ID, sender ID, recipient ID, the timestamp when it was sent, message subject, and message body. In a message thread with a series of back-and-forth messages, the initial message is the first message initiated by a sender, and the message ID of the initial message serves as the ID of the message thread. We then applied three filters: exclude the messages with empty message bodies; exclude the messages sent by mock-up patients and clinicians that were created for testing; and exclude messages sent by a clinician group where the sender uses a shared ID, usually for impersonal communication. In the end, we obtained a total of 5,654,514 secure messages sent by both patients and clinicians for the following analysis.
Message-Oriented and Sender-Oriented Analysis
We started by calculating the descriptive statistics for the approximately 5.6 million secure messages to probe four aspects of patient-clinician communication. (1) The total numbers (volumes) of messages sent by patients and clinicians and the overall counts of unique patient and clinician senders. We also distinguish whether a message is an initiated message from a sender (ie, a drug-related question from a patient or an appointment reminder from a clinician) or a replied message (ie, a follow-up question or clarification) in a message thread; (2) For patient-sent messages, we calculated the distribution of message counts per patient for the entire study period, the number of secure messages and the count of unique patient senders by year, and the distribution of message counts per patient each year. The patients who registered for patient portal but did not send any messages were excluded; (3) Similarly, for clinician-sent messages, we calculated the distribution of message counts per clinician for the entire study period, the number of secure messages and the count of clinician senders by year, and the distribution of message counts per clinician each year. In addition, we grouped the clinicians into 9 categories based on their practice roles (ie, physician, nurse practitioner/physician assistant [NP/PA], registered nurse [RN], and other) and care settings (ie, primary care, specialty, and other) as listed in, and measured the workload for each clinician category. The “Other—other” category refers to other supporting staffs who communicated with patients via secure messaging, such as patient appointment service specialists, social workers, and financial counselors who work outside of the primary and specialty care setting; and (4) To analyze the workload of clinicians in different times of a day, we split 24 hours into 12 time slices for each day (ie, 11 PM to 1 AM, 1 to 3 AM, 3 to 5 AM, 5 to 7 AM, 7 to 9 AM, 9 to 11 AM, 11 AM to 1 PM, 1 to 3 PM, 3 to 5 PM, 5 to 7 PM, 7 to 9 PM, and 9 to 11 PM) and calculated the percentage of secure messages that those clinicians sent in each of the 12 time slices by year.
aN/A: not applicable.
bNP: nurse practitioner.
cPA: physician assistant.
dRN: registered nurse.
We investigated two aspects of patient-clinician communication within message threads. The first aspect is the time delay of clinician responses to patient messages. The analysis of time delay between patient messages and clinician responses may suggest how promptly clinicians responded to patients. We identified all the pairs of patient-sent messages and clinician-replied messages in a message thread and calculated the time difference (in days) between them. We then calculated the distribution of time delays for the entire study period and for each year. Secondly, patients often communicate with clinicians back and forth for multiple times in a message thread. We examined these message threads by measuring the number of back-and-forth rounds in each message thread (ie, length of a message thread) and calculating the distribution of message threads in terms of message thread length over time.
We developed a series of scripts in Python (Python Software Foundation) together with popular Python libraries (eg, pandas , NumPy [ ], SciPy [ ], and Matplotlib [ ]) to perform data collection and preprocessing, statistical analysis, and visualization. No patients were exposed to any intervention. We used the data from the Mayo Clinic Unified Data Platform for analysis. The study was approved by the Mayo Clinic Institutional Review Board (19-002211).
Message-Oriented and Sender-Oriented Analysis
Descriptive Statistics of Secure Medical Messages and Their Senders
lists the total numbers of secure messages sent (ie, initiated and replied) by patients and clinicians as well as unique patient senders and clinician senders. The number of messages initiated by clinicians was almost identical to that of messages replied by clinicians but was slightly more than that of messages initiated by patients (1.7 million versus 1.5 million). However, the number of patient-initiated messages was about 3 times that of patient-replied messages (1.5 million versus 0.5 million). In addition, 93.7% of the patients (203,166/216,740) had initiated secure messages, whereas only 52.6% (113,974/216,740) had replied to their clinicians. This suggests that the patients in general initiated a message when they had health-related issues, rather than replying in a message thread, whereas the clinicians were obligated to initiate messages and respond to patient messages.
|Senders and secure message types||Numbers of secure messages, n (%)||Counts of unique senders, n (%)|
|Initiated messages||1,569,172 (74.06)||203,166 (93.74)|
|Replied messages||549,601 (25.94)||113,974 (52.59)|
|Initiated messages||1,774,000 (50.17)||5690 (67.27)|
|Replied messages||1,761,741 (49.83)||8,070 (95.40)|
The ratio of overall unique patient count to overall unique clinician count was about 25:1 (216,740 versus 8450). The ratio of patient count and clinician count per year increased from 7.7 (1919 versus 249) in 2010 to 21.0 (125,647 versus 5980) in 2017 as shown in Table S1 (). This indicates that the workload per clinician had an increasing trend over years in terms of patient users on the secure messaging system.
Patient-Sent Secure Messages
We illustrated the distribution of message count per patient, the total numbers of patient-sent messages and unique patients by year, and the box-whisker graph of message count per patient by year in(A-C). We found that 95.4% (206,818/216,740) of the patients sent less than 40 messages, as shown in (A). The median number of patient-sent messages was 4. The maximum number of patient-initiated and patient-replied messages was 2052 and 302, respectively (Table S1, ). We observed a similar increasing trend between the numbers of unique patients and patient-sent messages over years, as shown in (B), indicating the strong adoption of this technology. The Pearson correlation coefficient between the numbers of unique patients and patient-sent messages calculated with SciPy is r=1. The median of message count per patient increased from 1 to 3 during 2010-2017 as depicted in (C).
(D-F) depicts the distribution of message count per clinician, the total numbers of clinician-sent messages and unique clinicians by year, and the box-whisker graph of message count per clinician by year. In (D), the distribution of message count per clinician exhibits a “long tail” pattern, where the median is 77 but the max stretches far right at 18,314 (way beyond the X axis labeling range). Both the numbers of clinicians and clinician-sent messages were also increasing over time with the growths of the numbers of patients and patient-sent messages ( [E]). The Pearson correlation coefficient between the total numbers of clinician-sent messages and patient-sent messages over years is r=1, suggesting that the total number of clinician-sent message is strongly associated with that of patient-sent message. The median and IQR of message count per clinician steadily increased during 2010-2017 ( [F]). We observed that the count of clinicians largely increased but the median and IQR of message count per clinician decreased in 2013 compared with 2012 because many specialties at Mayo Clinic, Rochester started to use the patient portal.
Between 2010 and 2017, 19.93% (6773/8459) of the clinicians transitioned out of Mayo Clinic, Rochester and, therefore, we do not have information on their practice roles and care settings. We grouped the remaining 6773 clinicians, who generated 86.87% (3,071,529/3,535,741) messages, into 9 clinician categories. Based on these messages, we then analyzed the workload, message counts per clinician, and the distribution of messaging time in a day for each clinician category.
shows the total number of clinician-sent messages, the total number of clinicians, and the median of message count per clinician in each clinician category for each year. As shown in (A), the number of messages sent by clinicians in each clinician category was progressively increasing over time. Three clinician categories, namely “physician—primary care,” “Registered Nurse (RN)—specialty,” and “physician—specialty” had the largest increase in the generated messages. After 2014, the number of messages sent by these 3 clinician categories were over 2.5 times more than those of the other clinician categories combined. The category “physician—primary care” initiated the largest number of messages every year ( [B]). During 2010-2013, the “physician—primary care” category also replied the most to the messages from patients, but after 2013, “RN—specialty” took over the top spot in responding to patients ( [C]).
The total numbers of unique clinician senders in each category had a steady increase during 2010-2017 ([D-F]). After 2012, the clinician categories with top 3 message senders were “physician—specialty,” “other—specialty,” and “RN—specialty.” Most of the clinicians in the “other—specialty” category were responsible for responding to patient messages rather than initiating messages. The “nurse practitioner (NP) or physician assistant (PA)—primary care” category has the smallest size of clinician senders ( [D]).
In(G), the median of message count per clinician in 7 out of the 9 clinician categories steadily increased during 2010-2017. Among them, the top 2 clinician categories were “NP/PA—primary care” and “physician—primary care.” In 2017, the median of message count per clinician by the “NP/PA—primary care” category was over 2 times of any other groups ( [G-I]).
Clinician Messaging Workload Within a Day
We analyzed the clinician messaging time during a day across the 9 clinician categories in terms of message percentage () and message number (Figure S5-7, ). Most of the messages were sent from 7 AM to 5 PM. However, physicians and NP/PA also sent a considerable number of messages to patients between 5 PM and 7 PM. For example, the percentage of messages sent by “physician—primary care” during 5-7 PM increased from 7.95% (102/1283) in 2010 to 8.98% (2669/29,705) in 2012 and then dropped to 6.12% (15,714/256,761) in 2017. The percentage of messages sent by “NP/PA—primary care” during 5-7 PM remained constant around 5.08-6.75% (3/59-2,824/41,833), between 2010 and 2017.
We identified 1,576,205 message threads in 3,887,542 messages, which include 1,332,931 patient-initiated messages and 243,274 clinician-initiated messages, for thread-oriented analysis. Among these message threads, we identified 1,761,739 clinician responses to patient messages.
Time Delay Between Patient Messages and Clinician Responses
We illustrated the distribution of time delay between clinician responses and patient messages in. It appears that 96.0% (1,691,733/1,761,739) of patient messages were responded to by clinicians within 5 days. The shortest time a clinician spent responding to patients was 1 second, and the median was 0.6 days. With the increase in the total number of clinician messages and message count per clinician, the median time delay between clinician responses and patient messages increased from 0.13 days in 2010 to 0.59 days in 2014 but remained steady (0.53-0.59 days) after 2014. The IQR of time delay showed a trend similar to the median time delay over years.
Back-and-Forth Messages Between Patients and Clinicians in a Message Thread
We analyzed the distribution of message thread lengths (). We found that 95.1% (1,499,316/1,576,205) message threads had fewer than 5 back-and-forth messages. The median and maximum lengths of the message threads were 2 and 34, respectively. Between 2010 and 2017, the percentage of message threads with a length of 2 decreased from 88.0% (3429/3895) to 72.7% (372,424/512,395). The percentage of message threads with a length of 3 increased from 0.4% (16/3895) to 13.6% (69,754/512,395). The percentage of message threads with a length of 4 remained relatively stable, 6.1-9.6% (2354/38,537-374/3895). The percentage of the message threads with a length of 5 or more showed an uptrend over time. The median timespan of message threads was elongated as the message thread lengths increased.
Secure messaging in patient portal provides patients more convenient access for more personalized medical advice and information [, ]. Patients can send secure messages to their clinicians anywhere and anytime as they prefer, which fulfills a growing consumer model of medical care [ ]. Similar to previous results on secure messaging [ - ], our patient portal experienced a large increase in the numbers of patients, clinicians, and secure messages during 2010-2017. We observed a strong correlation between patient count and patient message count. The growth of patients and messages per patient both contributes to the upsurge of patient messages, suggesting a strong adoption of this health care technique and a growing pattern of patient engagement. The significant volume difference between patient-initiated messages and patient-replied messages (1.5 million vs 0.5 million) indicates the behavioral variation of patients for initiating and replying to messages; in general, the patients initiated a message when they encountered health issues instead of following an existing message thread. This implies potential opportunities to engage patients in communication and increase responses in pertinent contexts.
The clinician message volume is strongly associated with patient message volume because the clinicians were obligated to communicate with the patients for providing health care support. There were also practice-specific and role-specific patterns for the clinicians. For example, “physician—primary care” initiated the largest number of messages to patients every year, indicating their major role in patient engagement. After 2013, “RN—specialty” replied the most of messages to patients, suggesting that the patient portal opened up a longed-for channel of taking patient information needs in specialty care , and that specialty RNs represented the first-line taskforce to accommodate the needs.
The patient portal also manifested upward trends in the ratio of patient count to clinician count and numbers of messages per patient and clinician. The findings suggest an inflation of clinician workload in the secure messaging. Specifically, the “physician—primary care” and “NP/PA—primary care” categories had the heaviest per-clinician messaging workload every year because the “physician—primary care” category generated the largest number of clinician messages and the “NP/PA—primary care” category has the smallest size of clinician senders. This indicates where optimization is most needed in easing the clinician workload. Most (93.2%, 2,862,978/3,071,529) of the clinician messages were sent from 7 AM to 5 PM but 17.3% (532,894/3,071,529) of the messages were sent by clinicians to patients around noon (11 AM to 1 PM), and 5.94% (182,468/3,071,529) were sent after 5 PM (5-11 PM). Our findings were consistent with those from previous studies that secure messaging has a negative effect on clinician workload , and that about 24% of the work is carried out by clinicians on electronic health records for non–face-to-face care after clinician hours [ ].
Most of the clinicians responded to the patients in a timely manner. The median time delay of clinician responses to patient message remained constant (less than 0.60 days). Although the message volume and clinician workload persistently increased over years, the clinicians managed to respond to the patients in time. Regarding the message threads, the patients usually communicated with the clinicians in a few back-and-forth rounds. Possibly, patients and clinicians communicated via secure messaging for noncomplicated scenarios, or clinicians made a timely decision on a certain back-and-forth round about what acute issues and complex situations will require face-to-face visit or follow-up. The percentage of message threads with more than 2 rounds was rising, which may imply that the patients and clinicians were increasingly comfortable to communicate about more complex situations. However, further investigation is necessary to understand the potential mechanism of the messaging complexity of a communication thread.
The growing portal messages had different indications or impacts on its stakeholders. For the patients, a majority of them in this study received timely response to their concerns and requests. In the future, the patient engagement would probably continue to increase in terms of patient count and message count per patient, in particular, during and after the COVID-19 pandemic. For the clinicians, most of them think that secure messaging can have a positive effect on quality of care and patient safety, but secure messaging increased their burden of indirect patient care, as noted by Hoonakker et al . In some cases, clinicians need to send messages to patients after 5 PM in order to respond to them in time. With the rise of patient engagement and message volume over time [ - ], it will be critical to properly distribute the communication load for better efficiency and for avoiding clinician burnouts. For the health care system, patient portals and secure messaging may have a favorable impact on the cost-effectiveness of care [ ]. They could help to not only cut the administrative cost by alleviating the operational burden [ ], but also reduce patients’ health care utilization by improving their functional status [ ]. However, the detailed economical or cost-effectiveness analysis is beyond the scope of this paper, due to the difficulty of collecting relevant data. In the near future, some other health care systems will probably confront the management challenge of the fast-growing volume of patient messages [ ]. These health care systems would require new policies, billing models, or additional infrastructures. For example, more NP/PA, RN, and other support staff would be involved in replying to the increasing volume of patient messages. The artificial intelligence (AI) and natural language processing (NLP) tools would even be invested and developed to support the care teams for secure messaging [ - ].
Our study has several limitations. First, the secure messages were collected from Mayo Clinic, Rochester, a tertiary care institution for complex medical conditions, and might not be representative of different patient populations or clinical settings in other parts of the country. Second, the patient secure messages after 2017 were not included in this study due to the upgrade of the patient portal system at Mayo Clinic in 2018. The COVID-19 pandemic is transforming the health care delivery via telehealth, and we would investigate the secure messaging after 2017 and the impact of the COVID-19 on the patient portal system for a future study. Third, the 9 clinician categories we used were not always accurate because a small portion of clinicians changed their practice roles and care settings during the study period. Fourth, there was no guarantee that the clinician-initiated or clinician-replied messages were always written by themselves. Finally, linking patient secure messages to other patient medical records for a more detailed study of the needs of different patient populations is out of the scope of this study, but represents an important area we would like to explore in the future.
We performed message-oriented, sender-oriented, and thread-oriented analyses to probe and characterize millions of secure medical messages generated by patients and clinicians with diverse backgrounds. We analyzed the message volumes, patient or clinician population sizes, message counts per patient or clinician, and their trends over years. We computed the time distribution of clinician messaging to further understand their workload in different time slices of a day. For each message thread, we calculated the time delay between patient messages and clinician responses to examine the responding efficiency and the number of back-and-forth rounds to roughly assess the communication complexity.
Our study shows a steady rise in patient involvement, through the use of secure messaging, and workload per clinician over years. However, most clinicians were responding to the patients in a timely manner in order to meet their needs. Our findings shed light on opportunities for care teams to improve messaging tasks and optimize clinician workload and for the experts in AI and NLP to develop robust and intelligent messaging tools to support the care teams for better communication efficiency and quality. These findings offer valuable information on the digital interaction between patients and clinicians and may serve as a reference for promoting patient-centered care.
In the future, we will perform a content analysis of patient secure messages using AI and NLP and examine the patient populations in terms of socioeconomic factors by linking them to patient medical records. We will also perform a comparative study between patient portal messages and traditional health care services and a survey study to understand patient and clinician experiences on using patient portal messaging.
Funding for this study was provided by the Mayo Clinic Center for Clinical and Translational Science (UL1TR002377) and the National Library of Medicine (5K01LM012102).
LY designed and guided the research study, discussed the methods and results, and revised the manuscript. MH preprocessed the data, implemented the algorithms, performed the computations and analyses, and drafted and revised the manuscript. JF discussed the methods and results and revised the manuscript. JP, NDS, and BAC participated in the discussion and provided feedback on the manuscript.
Conflicts of Interest
Supplementary materials.DOCX File , 930 KB
- Irizarry T, DeVito Dabbs A, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet Res 2015 Jun 23;17(6):e148 [FREE Full text] [CrossRef] [Medline]
- Tieu L, Sarkar U, Schillinger D, Ralston JD, Ratanawongsa N, Pasick R, et al. Barriers and Facilitators to Online Portal Use Among Patients and Caregivers in a Safety Net Health Care System: A Qualitative Study. J Med Internet Res 2015 Dec 03;17(12):e275 [FREE Full text] [CrossRef] [Medline]
- Centers for Medicare & Medicaid Services (CMS), HHS. Medicare and Medicaid programs; electronic health record incentive program--stage 2. Final rule. Fed Regist 2012 Sep 04;77(171):53967-54162 [FREE Full text] [Medline]
- Henry J, Barker W, Kachay L. Electronic Capabilities for Patient Engagement among U.S. Non-Federal Acute Care Hospitals: 2012-2015. The Office of the National Coordinator for Health Information Technology. 2019. URL: https://www.healthit.gov/sites/default/files/briefs/2015_patient_engagement_data_brief.pdf [accessed 2021-12-10]
- Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015 Feb 10;17(2):e44 [FREE Full text] [CrossRef] [Medline]
- Wade-Vuturo AE, Mayberry LS, Osborn CY. Secure messaging and diabetes management: experiences and perspectives of patient portal users. J Am Med Inform Assoc 2013 May 01;20(3):519-525 [FREE Full text] [CrossRef] [Medline]
- Osborn CY, Mayberry LS, Wallston KA, Johnson KB, Elasy TA. Understanding patient portal use: implications for medication management. J Med Internet Res 2013 Jul 03;15(7):e133 [FREE Full text] [CrossRef] [Medline]
- Horvath M, Levy J, L'Engle P, Carlson B, Ahmad A, Ferranti J. Impact of health portal enrollment with email reminders on adherence to clinic appointments: a pilot study. J Med Internet Res 2011 May 26;13(2):e41 [FREE Full text] [CrossRef] [Medline]
- Guy R, Hocking J, Wand H, Stott S, Ali H, Kaldor J. How effective are short message service reminders at increasing clinic attendance? A meta-analysis and systematic review. Health Serv Res 2012 Apr 08;47(2):614-632 [FREE Full text] [CrossRef] [Medline]
- Zhou YY, Garrido T, Chin HL, Wiesenthal AM, Liang LL. Patient access to an electronic health record with secure messaging: impact on primary care utilization. Am J Manag Care 2007 Jul;13(7):418-424 [FREE Full text] [Medline]
- Simon GE, Ralston JD, Savarino J, Pabiniak C, Wentzel C, Operskalski BH. Randomized trial of depression follow-up care by online messaging. J Gen Intern Med 2011 Jul 8;26(7):698-704 [FREE Full text] [CrossRef] [Medline]
- Hoonakker PL, Carayon P, Cartmill RS. The impact of secure messaging on workflow in primary care: Results of a multiple-case, multiple-method study. Int J Med Inform 2017 Apr;100:63-76 [FREE Full text] [CrossRef] [Medline]
- Arndt BG, Beasley JW, Watkinson MD, Temte JL, Tuan W, Sinsky CA, et al. Tethered to the EHR: Primary Care Physician Workload Assessment Using EHR Event Log Data and Time-Motion Observations. Ann Fam Med 2017 Sep 11;15(5):419-426 [FREE Full text] [CrossRef] [Medline]
- Crotty BH, Tamrat Y, Mostaghimi A, Safran C, Landon BE. Patient-to-physician messaging: volume nearly tripled as more patients joined system, but per capita rate plateaued. Health Aff (Millwood) 2014 Oct;33(10):1817-1822 [FREE Full text] [CrossRef] [Medline]
- Cronin R, Davis S, Shenson J, Chen Q, Rosenbloom S, Jackson G. Growth of Secure Messaging Through a Patient Portal as a Form of Outpatient Interaction across Clinical Specialties. Appl Clin Inform 2017 Dec 19;06(02):288-304. [CrossRef]
- Shenson JA, Cronin RM, Davis SE, Chen Q, Jackson GP. Rapid growth in surgeons' use of secure messaging in a patient portal. Surg Endosc 2016 Apr 27;30(4):1432-1440 [FREE Full text] [CrossRef] [Medline]
- Masterman M, Cronin R, Davis S, Shenson J, Jackson G. Adoption of Secure Messaging in a Patient Portal across Pediatric Specialties. AMIA Annu Symp Proc 2017;2016:1930-1939 [FREE Full text] [Medline]
- Wosik J, Fudim M, Cameron B, Gellad Z, Cho A, Phinney D, et al. Telehealth transformation: COVID-19 and the rise of virtual care. J Am Med Inform Assoc 2020 Jun 01;27(6):957-962 [FREE Full text] [CrossRef] [Medline]
- Patient Online Services. 2021. URL: https://onlineservices.mayoclinic.org/content/staticpatient/showpage/patientonline [accessed 2021-12-03]
- McKinney W. Data structures for statistical computing in python. 2010 Jun 1 Presented at: Proceedings of the 9th Python in Science Conference; 2010; Austin, Texas. [CrossRef]
- Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, et al. Array programming with NumPy. Nature 2020 Sep 16;585(7825):357-362 [FREE Full text] [CrossRef] [Medline]
- Virtanen P, Gommers R, Oliphant T, Haberland M, Reddy T, Cournapeau D, SciPy 1.0 Contributors. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods 2020 Mar;17(3):261-272 [FREE Full text] [CrossRef] [Medline]
- Hunter JD. Matplotlib: A 2D Graphics Environment. Comput. Sci. Eng 2007 May;9(3):90-95. [CrossRef]
- Coughlin S, Prochaska J, Williams LB, Besenyi G, Heboyan V, Goggans S, et al. Patient web portals, disease management, and primary prevention. RMHP 2017 Apr;10:33-40. [CrossRef]
- Chen AH, Yee HF. Improving the primary care-specialty care interface: getting from here to there. Arch Intern Med 2009 Jun 08;169(11):1024-1026. [CrossRef] [Medline]
- Riippa I, Linna M, Rönkkö I. A Patient Portal With Electronic Messaging: Controlled Before-and-After Study. J Med Internet Res 2015 Nov 09;17(11):e250 [FREE Full text] [CrossRef] [Medline]
- Kamo N, Bender AJ, Kalmady K, Blackmore CC. Meaningful use of the electronic patient portal - Virginia Mason's journey to create the perfect online patient experience. Healthc (Amst) 2017 Dec;5(4):221-226. [CrossRef] [Medline]
- Sorondo B, Allen A, Fathima S, Bayleran J, Sabbagh I. Patient Portal as a Tool for Enhancing Patient Experience and Improving Quality of Care in Primary Care Practices. EGEMS (Wash DC) 2016 Jan 26;4(1):1262 [FREE Full text] [CrossRef] [Medline]
- Sulieman L, Gilmore D, French C, Cronin RM, Jackson GP, Russell M, et al. Classifying patient portal messages using Convolutional Neural Networks. J Biomed Inform 2017 Oct;74:59-70 [FREE Full text] [CrossRef] [Medline]
- Sulieman L, Robinson JR, Jackson GP. Automating the Classification of Complexity of Medical Decision-Making in Patient-Provider Messaging in a Patient Portal. J Surg Res 2020 Nov;255:224-232 [FREE Full text] [CrossRef] [Medline]
- De A, Huang M, Feng T, Yue X, Yao L. Analyzing Patient Secure Messages Using a Fast Health Care Interoperability Resources (FIHR)-Based Data Model: Development and Topic Modeling Study. J Med Internet Res 2021 Jul 30;23(7):e26770 [FREE Full text] [CrossRef] [Medline]
|AI: artificial intelligence|
|NLP: natural language processing|
|NP: nurse practitioner|
|PA: physician assistant|
|RN: registered nurse|
Edited by R Kukafka; submitted 13.05.21; peer-reviewed by A Barker, J Ritchie, G Cenikj; comments to author 28.06.21; revised version received 23.08.21; accepted 18.11.21; published 11.01.22Copyright
©Ming Huang, Jungwei Fan, Julie Prigge, Nilay D Shah, Brian A Costello, Lixia Yao. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 11.01.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.