Published on 29.08.11 in Vol 13, No 3 (2011): Jul-Sep
An Evaluation of the Use of Smartphones to Communicate Between Clinicians: A Mixed-Methods Study
Background: Communication between clinicians is critical to providing quality patient care but is often hampered by limitations of current systems. Smartphones such as BlackBerrys may improve communication, but studies of these technologies have been limited to date.
Objective: Our objectives were to describe how smartphones were adopted for clinical communication within general internal medical wards and determine their impact on team effectiveness and communication.
Methods: This was a mixed-methods study that gathered data from the frequency of smartphone calls and email messages, clinicians' interviews, and ethnographic observations of clinical communication interactions. Triangulation of qualitative and quantitative data was undertaken to develop common themes that encompass comprehensive and representative insights across different methods.
Results: Findings from our study indicated that over a 24-hour period, nurses sent on average 22.3 emails to the physicians mostly through the “team smartphone,” the designated primary point of contact for a specific medical team. Physicians carrying the team smartphone received on average 21.9 emails and 6.4 telephone calls while sending out 6.9 emails and initiating 8.3 telephone calls over the 24-hour period. Our analyses identified both positive and negative outcomes associated with the use of smartphones for clinical communication. There was a perceived improvement in efficiency over the use of pagers for clinical communication for physicians, nurses, and allied health professionals. In particular, residents found that the use of smartphones helped to increase their mobility and multitasking abilities. Negative outcomes included frequent interruptions and discordance between what doctors and nurses considered urgent. Nurses perceived a worsening of the interprofessional relationships due to overreliance on messaging by text with a resulting decrease in verbal communication. Unprofessional behaviors were observed in the use of smartphones by residents.
Conclusions: Routine adoption of smartphones by residents appeared to improve efficiency over the use of pagers for physicians, nurses, and allied health professionals. This was balanced by negative communication issues of increased interruptions, a gap in perceived urgency, weakened interprofessional relationships, and unprofessional behavior. Further communication interventions are required that balance efficiency and interruptions while maintaining or even improving interprofessional relationships and professionalism.
J Med Internet Res 2011;13(3):e59
Effective communication between clinicians to coordinate patient care is critical for providing quality health care to patients . Frequent interruptions through paging is a major communication issue [ - ], and poor communication can result in inefficiencies and errors [ - ]. The burden from inefficient communication has been well documented in multiple areas of hospital care, and a systematic review has linked interruptions to medical errors [ - ].
The use of smartphones such as BlackBerrys may improve hospital communication as they provide multiple communication modalities . For example, with urgent issues, direct calling can eliminate the need to wait for a page to be answered. In contrast, for nonurgent issues, asynchronous communication through email can be used, which could reduce disruptions. While many clinicians own cellular phones and smartphones, their use for clinical communication is variable [ , ]. Other than two studies that reported a perceived improvement in workflow efficiency from clinicians’ surveys, the study of smartphones used in hospital communication has been limited [ , ].
To understand the impact of the use of smartphones on the delivery of hospital care, we conducted a mixed-methods study to describe how smartphones were used, to identify advantages and disadvantages associated with their use, and to determine how their use can be improved.
Beginning in March 2008, each resident on the general internal medicine units received an individual BlackBerry smartphone to use for clinical communication typically within or among the medical teams . BlackBerry devices were selected because of the secure email functionality and because these were the standard smartphones used by hospital administration. In addition, each team also had a “team BlackBerry” that was designated as the primary point of contact for nurses and allied health professionals to communicate with the teams. The team BlackBerry was typically carried during the day by the senior resident and then given to the covering junior resident during sign over. Specifically, nurses would contact residents by sending emails to the team BlackBerry with the following structures and information: (1) the patient’s name, (2) the nurse’s name, (3) the issue and purpose of contact, and (4) their preferred response (callback, email, or no response required). For urgent patient issues, nurses and other clinicians were asked to call the team BlackBerry directly. For nonurgent issues, clinicians were asked to contact the team BlackBerry through email. This recommendation was based on previous findings in which a high number of direct calls were found to be very disruptive for residents [ ].
A mixed-methods approach was adopted to obtain a variety of data sources on the communication processes occurring on the wards. The following three methods were used: (1) quantitative measures assessing the frequency and use of smartphone calls and email messaging, (2) semistructured interviews with clinicians, and (3) ethnographic observations of clinical communication interactions were conducted using two techniques: (1) nonparticipatory “work shadowing” (defined below) and (2) observations at the nursing stations.
The settings were four general internal medicine wards at two large urban teaching hospitals in Canada. At each site, there were four medical teams each consisting of an attending physician, a senior resident, junior residents, and medical students. The study was conducted from January 5, 2009, until May 28, 2010.
Between January 5, 2009 and May 28, 2010, quantitative and qualitative data were collected on communication patterns from in-depth interviews and from observations of communications.
Quantitative data on the usage of the smartphones to receive and place telephone calls and emails were gathered by accessing email accounts and phone records of consenting residents. A total of 12,936 emails and 13,717 phone calls were analyzed from 34 residents ().
Semistructured interviews were conducted to explore clinicians’ perceptions of their experiences using the smartphones. To conduct qualitative comparisons and to ensure that a variety of clinicians’ perspectives were represented, we adopted a purposive sampling strategy where different groups of health care professionals with differing views on the use of smartphones for clinical communications were interviewed (). Each interview lasted between 15 and 40 minutes and was carried out at a mutually convenient location within the two hospital sites. The interviews were conducted and audio taped by an independent research associate and then professionally transcribed before analysis. The interview protocol consisted of a series of open-ended questions with appropriate follow-up probes that focused on users’ perspectives and their experiences using the smartphone technology.
Observations of Communications
Ethnographic observational methods were employed to explore and understand the in-depth communication processes and behaviors around smartphone use . Field notes included recording the usage on the types of communication tools used as well as communication interactions and incidents between clinicians. Two types of observational methods were used. First, a nonparticipatory “work-shadowing” approach was employed whereby a researcher followed medical residents for 2- to 5-hour periods during the day and evening shifts starting from 10 am to 11 pm. Both incoming and outgoing communications were recorded during these shifts. Second, observations were also conducted for 2-hour periods at the general internal medicine nursing stations, the hubs of interprofessional communication starting from 10 am and lasting until 10 pm ( ).
The study was approved by the Research Ethics Board, University Health Network, Toronto, Ontario.
To determine communication volume using smartphones, descriptive statistics of calls and emails per day were calculated from email and call logs. Incoming and outgoing communications recorded during work-shadowing sessions were also identified and descriptive statistics of the different communication methods per hour were calculated. All emails sent by the consenting residents were analyzed to determine the frequency of emails that were received by others such as nurses, attending physicians, or allied health providers. Similarly, all emails received by consenting residents were analyzed to determine the frequency of emails received from different types of senders.
Interviews were transcribed, and inductive thematic analysis was performed using the qualitative data analysis software NVivo 8 (QSR international, Doncaster, Victoria, Australia). The transcripts were coded by three members of the research team (authors RW, SR, and VL) to derive and identify a number of common perceptions and broad themes.
For the ethnographic methods, the number and types of communication events occurring during the work-shadowing sessions were tabulated. Field notes from both the work-shadowing and ward observations were reviewed to identify common themes.
Triangulation of qualitative and quantitative data was undertaken to develop themes that encompass comprehensive and representative insights that are common across multiple methods.
Key results are presented in two main sections. First quantitative data reporting smartphone communication volume is described. Second, qualitative data describing perceptions and ethnographic fieldwork is presented.
Description of Communication Volume
The usage of smartphones for telephone calls is shown in. Outgoing calls were placed to the hospital 41.2% of the time, to another BlackBerry 25.3% of the time, and to external numbers 33.4% of the time. From the 12,936 emails that were sent from or received from the smartphones, the daily frequencies of who communicates with whom were calculated ( ).
The different incoming and outgoing communications observed during work shadowing of residents are listed in. Note that these numbers are different from those calculated from device usage as work shadowing typically occurred at busier times during the day and evening.
The analysis of qualitative data (interviews, work shadowing, and ward observations) generated five major themes: efficiency, interruptions, interprofessional relations, gaps in perceived urgency, and professionalism ().
Nurses and Allied Health Professionals
With the smartphone system, nurses no longer felt the need to wait for a telephone reply, which typically was required with paging, and this resulted in less “phone tag.” Some nurses and allied health professionals perceived a faster response and increased accessibility to physicians (, data extract 1). They found the use of emails helped to convey their patient’s status quickly and efficiently to doctors ( , data extract 2). Nurses also reported that since their emails were sent to the team BlackBerry, less time was spent trying to locate a specific resident ( , data extract 3). Some of the nurses interviewed appreciated that the smartphone system allowed direct and immediate communication by phone with physicians for urgent issues. Direct calls were observed to be from nurses or other clinicians who previously were unsuccessful at getting a response from a resident through email or paging ( , data extract 4).
A number of the nurses interviewed perceived that the new system added a barrier, with increased difficulty reaching doctors. Instead of being able to resolve complex issues quickly with a page coupled with a brief discussion over a telephone call as done in the old system, nurses found it unproductive to have multiple emails being sent back and forth (, data extract 5). Instead, nurses found that telephone or face-to-face conversations allowed more detailed discussions compared with the short text of an email.
Residents also perceived significant efficiency with the use of smartphones. Since 42% of emails from nurses were informational items, no follow up response was required for these types of communication (, data extract 6). Residents also used the smartphones to reduce the inefficiencies of having to page other services to a ward telephone. By paging other physicians to their smartphone, they no longer needed to wait at a ward telephone for a callback and were able to perform other tasks while waiting for the return call ( , data extract 7). To the residents, smartphones made it easier to coordinate activities within the teams through email or telephone calls ( , data extract 8). The devices were also used to communicate with other physicians from other teams and services to increase communication around patients ( , data extract 9). Residents appeared to incorporate the phone and messaging functions of the smartphone in their clinical work, using them to call a patient's family member, to communicate rapidly with other team members, and to respond to urgent situations ( , data extract 10).
Despite residents acknowledging that the use of smartphones had reduced the time spent on responding to informational messages, residents also perceived an increase in the overall number of messages and calls received. The substantial volume of interruptions was reflected in the number of phone calls and emails received as determined by both smartphone usage as well as by work shadowing residents (). From analyzing the frequency of calls and emails, a senior resident who usually carried both the team BlackBerry (point of contact and communication for nursing and allied health staff) and the senior BlackBerry (typically used for communication within the medical teams) would receive on average 11.6 calls and 25 emails within a 24-hour period. The volume of calls and emails could be higher during evenings and weekends when residents were covering for other teams and thus carrying multiple smartphones. ( , data extract 1).
Interviews with the staff physicians also highlighted similar observations where physicians noticed the disruptions that the smartphone brought upon their teaching and patient care rounds (, data extract 2). From work-shadowing sessions, there were 4.6 average interruptions per hour for a resident when considering all direct calls, emails, and face-to-face communications ( ). These interruptions, however, were sometimes even more frequent. For example, within a 40-minute team meeting with the attending physician, 7 interruptions were observed. This included 5 direct phone calls to residents, where residents ended up taking the calls and leaving the room while the meeting was in progress ( , data extract 3). In particular, numerous patient interactions were noted to be interrupted by direct calls. Often residents were observed answering phone calls on their smartphones, exiting the room, and then resuming patient interaction once the call was completed ( , data extract 4). Residents did note that this had a negative impact on communication, especially during family meetings.
Considering these high volumes of communications, it is not surprising that residents commented that they felt overwhelmed by the constant interruptions, which may be a result of the increased availability of multiple communication options (, data extracts 5-7). Similar sentiments were echoed by the staff physicians, who observed that these interruptions could have detrimental impacts such as reduced downtime and impede residents’ abilities to provide patient care ( , data extract 8).
A strong theme that emerged throughout interviews was the impact of this new technology on interprofessional relationships. Nurses commented that the new system reduced opportunities for face-to-face interactions, which many valued. Nurses reported that they found it more difficult to build interprofessional relationships through the new technology. Specifically, the smartphones and use of email messaging reduced verbal conversations, which nurses felt prevented them from getting to know the residents, discouraged interest in their work, and reduced opportunities for nurses to have direct educational experiences with the residents (, data extracts 1 and 2). Additionally, nurses found the process depersonalizing to have the team BlackBerry as the primary point of contact instead of having direct interactions with the specific physician ( , data extracts 2 and 3).
In contrast, physicians perceived no major negative changes with this new technology. Since nurses were required to type in their names to send emails, physicians felt that they learned nurses’ names better and thus perceived their interprofessional relationships had actually improved (, data extracts 4 and 5).
Gap in Perceived Urgency
From our data analyses, we identified a gap in what physicians and nurses perceived as urgent patient issues. If a physician did not perceive the issue communicated to be urgent, often there would not be a response despite a request to respond having been made by the nurse, or the resident would reply with an email when the nurse requested a telephone call or otherwise.
Nurses perceived both a lack of acknowledgment of messages as well as not receiving the requested response (, data extracts 1-3). This perception was confirmed by analyzing responses to nurses’ emails in which nurses actually only received an email response 50% of the time requested. They also felt many physicians ignore emails, similar to their ignoring pages previously. Nurses found it frustrating and felt belittled when physicians ignored their communications ( , extract 4). With a lack of acknowledgment, nurses often felt the need to resend messages. Nurses also reported the need to have clear specifications of when and how to inform physicians such as for abnormal vital signs or laboratory results. It was observed that nurses would consult each other to see if physicians should be informed ( , data extract 5).
Physicians also commented that there were too many direct calls for low importance items and a high number of emails that were of trivial importance (, data extract 6).
Much of this gap of perceived urgency may be attributed to the numerous methods that are now available to contact physicians. Though the smartphone system provides various options to contact residents, the array of choices often created confusion and a mismatch of responses among clinicians (, data extract 7).
This theme focused on physicians’ behavior around the use of smartphones and how nurses, allied health professionals, and attending physicians perceived the manner in which residents handled interruptions by phone call or email message that could be regarded as unprofessional.
While having a conversation with a resident on the ward, nurses and other allied health professionals occasionally reported that the resident would answer a call on his or her smartphone, thus interrupting their conversation. Nursing and allied health professionals also found it disruptive when residents answered calls during interprofessional rounds (, data extracts 1 and 2).
Similarly, during patient rounds or educational rounds, attending physicians mentioned that they also found residents’ behavior with smartphones to be unprofessional at times (, data extracts 3 and 4). One attending physician observed that while smartphones increased the availability of the residents to other clinicians, it reduced the local availability of residents due to the constant distractions and interruptions from the device. In effect, the residents were made more global but less local ( , data extracts 5 and 6). As noted by some, the continued calls and checking of messages often took away the quality time which residents spent interacting with the attending physician and their interprofessional colleagues.
Finally, this behavior was also observed during patients’ interactions, where residents would pick up phone calls, check, or type messages while talking with patients or supervising a procedure. Although patients’ perceptions were not obtained, other clinicians commented that such behavior could create negative perceptions among patients (, data extracts 7 and 8).
In this study, we described the nature of communications and the perceived impact of a new smartphone system for residents on busy general medical services. Smartphones were used frequently by residents to communicate about patients. We found that residents, nurses, and other clinicians perceived improvements in efficiency over the traditional paging system likely because smartphones appear to address many of the known issues with numeric paging such as inability to triage information or having to wait by a phone for a return call. The overall number of interruptions, however, was perceived to have increased likely because it was now easier for nurses to initiate communications. With more communications occurring over written modes such as emails and text messages, nurses perceived a negative impact to relationships with physicians because of decreased verbal conversations. Smartphone use in this study also highlighted the discordance between what nurses and physicians perceive as urgent. Finally, residents’ behavior when communicating with smartphones during patient care and education learning activities was perceived at times as unprofessional.
While the gap between what nurses and physicians perceive to be urgent has been previously identified , we found that the gap was likely made greater by increasing the number of methods for clinical communication. Email communication may also decrease verbal interactions—deemed by many clinicians as the most effective and optimal method of communication—which can result in the impediments of interprofessional collaboration and relationships [ ]. Finally, the issue of digital professionalism is part of a larger issue of both knowing and educating medical trainees in the proper use of new media [ , ]. While there are established definitions of medical professionalism, these have not necessarily kept pace with the rapidly evolving new digital media [ - ]. Without being given formal guidance in using this new technology, residents appear to handle interruptions by trying to be efficient and minimize response times but at the expense of interprofessional and patient relationships.
Our findings highlight the important aspects to consider when implementing systems to improve clinical communication. While increasing overall team communication and efficiency are important goals, it is also equally critical to consider the range of possible and potential impacts as unintended consequences can occur (). Considering these themes (efficiency, interruptions, interprofessional relationships, common understanding of urgency, and professionalism) as a framework could aid in the design and evaluation of communication systems. This may facilitate the development of communication systems in which the appropriate information is sent through the appropriate medium with the appropriate intrusiveness, while incorporating processes that foster interprofessional relationships and promote digital professionalism.
As shown in, our findings suggest that email or text conversations appear to create the desirable and undesirable effects as listed above. The increased modes of communication when moving from pagers to smartphones include both text messages as well as direct phone calls. Our findings support that increased modes of communication appear to highlight the gap in perceived urgency and highlights professionalism issues.
Comparison With Other Work
The issues with numeric paging have been well documented. These include a high number of interruptions with an average of 9 pages per hour on some clinical services . While all pages interrupt, only 30% have been found to require urgent attention, and the majority do not require a response within an hour [ ]. As well, one study found that 14% of pages are sent to the wrong physician, with 47% of those pages requiring urgent attention [ ]. The response rate for pages in one observational study was 90% [ ].
To our knowledge, this is the first comprehensive mixed-methods evaluation of the use of smartphones on hospital communications. Studies using surveys have found perceived improvements with the use of smartphones among clinicians [, ]. Through the use of ethnography observations and interviews, we were able to determine how smartphones are used and their impacts on different domains including unintended effects.
There were limitations to this study. Smartphones were provided only to residents, and the behavior and perceptions would likely be different if other professions were provided with smartphones. The study was also conducted at two hospital sites; thus, generalizing to other institutions with different hospital cultures may yield different results. However, our intervention used standard components of smartphones and email with minimal customization, and other academic hospitals may be able to learn from this experience. Future studies could look at other measurements, outcomes, and impacts on quality of patient care such as time to resolution of urgent items or the effect of interruptions on patient care.
Routine adoption of smartphones by residents appeared to improve efficiency over the use of pagers for physicians, nurses, and allied health professionals. This was balanced by negative communication issues of increased interruptions, a gap in perceived urgency, weakened interprofessional relationships, and unprofessional behavior. Further communication interventions are required that balance efficiency and interruptions while maintaining or even improving interprofessional relationships and professionalism.
We would like to thank the Ontario Ministry of Health and Long Term Care Alternate Funding Proposal Innovation Grant for providing funding. We would like to thank Christine Plaza for help with designing the figures.
Conflicts of Interest
- Committee on Quality of Health Care in America, Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
- Rivera-Rodriguez AJ, Karsh BT. Interruptions and distractions in healthcare: review and reappraisal. Qual Saf Health Care 2010 Aug;19(4):304-312. [CrossRef] [Medline]
- Coiera E. When conversation is better than computation. J Am Med Inform Assoc 2000;7(3):277-286. [Medline]
- Harvey R, Jarrett PG, Peltekian KM. Patterns of paging medical interns during night calls at two teaching hospitals. CMAJ 1994 Aug 1;151(3):307-311. [Medline]
- Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991 Feb 7;324(6):370-376 [FREE Full text] [CrossRef] [Medline]
- Wilson RM, Runciman WB, Gibberd RW, Harrison BT, Hamilton JD. Quality in Australian Health Care Study. Med J Aust 1996 Jun 17;164(12):754. [Medline]
- Wong BM, Quan S, Cheung CM, Morra D, Rossos PG, Sivjee K, et al. Frequency and clinical importance of pages sent to the wrong physician. Arch Intern Med 2009 Jun 8;169(11):1072-1073. [CrossRef] [Medline]
- Collins S, Currie L, Bakken S, Cimino JJ. Interruptions during the use of a CPOE system for MICU rounds. AMIA Annu Symp Proc 2006:895. [Medline]
- Collins S, Currie L, Patel V, Bakken S, Cimino JJ. Multitasking by clinicians in the context of CPOE and CIS use. Stud Health Technol Inform 2007;129(Pt 2):958-962. [Medline]
- Brixey JJ, Tang Z, Robinson DJ, Johnson CW, Johnson TR, Turley JP, et al. Interruptions in a level one trauma center: a case study. Int J Med Inform 2008 Apr;77(4):235-241. [CrossRef] [Medline]
- Laxmisan A, Hakimzada F, Sayan OR, Green RA, Zhang J, Patel VL. The multitasking clinician: decision-making and cognitive demand during and after team handoffs in emergency care. Int J Med Inform 2007;76(11-12):801-811. [CrossRef] [Medline]
- Hickam DH, Severance S, Feldstein A, Ray L, Gorman P, Schuldheis S, et al. The effect of health care working conditions on patient safety. Evid Rep Technol Assess (Summ) 2003 Mar(74):1-3. [Medline]
- Soto RG, Chu LF, Goldman JM, Rampil IJ, Ruskin KJ. Communication in critical care environments: mobile telephones improve patient care. Anesth Analg 2006 Feb;102(2):535-541 [FREE Full text] [CrossRef] [Medline]
- Derbyshire SW, Burgess A. Use of mobile phones in hospitals. BMJ 2006 Oct 14;333(7572):767-768. [CrossRef] [Medline]
- Wu RC, Morra D, Quan S, Lai S, Zanjani S, Abrams H, et al. The use of smartphones for clinical communication on internal medicine wards. J Hosp Med 2010;5(9):553-559. [CrossRef] [Medline]
- O'Connor C, Friedrich JO, Scales DC, Adhikari NK. The use of wireless e-mail to improve healthcare team communication. J Am Med Inform Assoc 2009;16(5):705-713. [CrossRef] [Medline]
- Reeves S, Kuper A, Hodges BD. Qualitative research methodologies: ethnography. BMJ 2008;337:a1020. [Medline]
- Coiera E, Tombs V. Communication behaviours in a hospital setting: an observational study. BMJ 1998 Feb 28;316(7132):673-676 [FREE Full text] [Medline]
- Kock N. The ape that used email: Understanding e-communication behavior through evolution theory. Commun Assoc Inf Syst 2001;5(3):1-29.
- Parikh SM, Liu E, White CB. Connectivity need not come at the expense of professionalism. Acad Med 2010 Jun;85(6):930; author reply 930. [CrossRef] [Medline]
- Spiegelman J, Detsky AS. Instant mobile communication, efficiency, and quality of life. JAMA 2008 Mar 12;299(10):1179-1181. [CrossRef] [Medline]
- Frank JR, Danoff D. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach 2007 Sep;29(7):642-647. [CrossRef] [Medline]
- Swick HM. Toward a normative definition of medical professionalism. Acad Med 2000 Jun;75(6):612-616. [Medline]
- ABIM Foundation. American Board of Internal Medicine, ACP-ASIM Foundation. American College of Physicians-American Society of Internal Medicine, European Federation of Internal Medicine. Medical professionalism in the new millennium: a physician charter. Ann Intern Med 2002;136(3):243-246 [FREE Full text] [Medline]
- Jain SH. Practicing medicine in the age of Facebook. N Engl J Med 2009 Aug 13;361(7):649-651 [FREE Full text] [CrossRef] [Medline]
- Chiu T, Old A, Naden G, Child S. Frequency of calls to "on-call" house officer pagers at Auckland City Hospital, New Zealand. N Z Med J 2006;119(1231):U1913. [Medline]
- Patel R, Reilly K, Old A, Naden G, Child S. Appropriate use of pagers in a New Zealand tertiary hospital. N Z Med J 2006;119(1231):U1912. [Medline]
Edited by G Eysenbach; submitted 24.09.10; peer-reviewed by J Ai, P Chhanabhai; comments to author 20.02.11; revised version received 21.07.11; accepted 21.07.11; published 29.08.11
©Robert Wu, Peter Rossos, Sherman Quan, Scott Reeves, Vivian Lo, Brian Wong, Mark Cheung, Dante Morra. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 29.08.2011.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.