Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 07.03.19 in Vol 21, No 3 (2019): March

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/11123, first published May 24, 2018.

This paper is in the following e-collection/theme issue:

    Original Paper

    Evaluation of an E-Learning Training Program to Support Implementation of a Group-Based, Theory-Driven, Self-Management Intervention For Osteoarthritis and Low-Back Pain: Pre-Post Study

    1School of Public Health, Physiotherapy and Sports Science, University College Dublin, Dublin, Ireland

    2Faculty of Medicine, Memorial University, St Johns, NL, Canada

    3Centre for Rehabilitation Research in Oxford (RRiO), Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom

    4School of Psychology, University College Dublin, Dublin, Ireland

    5Faculty of Health, Arts and Design, Swinburne University of Technology, Melbourne, Australia

    *these authors contributed equally

    Corresponding Author:

    Deirdre A Hurley, BPhysio, MAppSc, PhD

    School of Public Health, Physiotherapy and Sports Science

    University College Dublin

    Belfield

    Dublin, D4

    Ireland

    Phone: 353 17166524

    Fax: 353 17166501

    Email: deirdre.hurleyosing@ucd.ie


    ABSTRACT

    Background: By adaptation of the face-to-face physiotherapist-training program previously used in the Self-management of Osteoarthritis and Low back pain through Activity and Skills (SOLAS) feasibility trial, an asynchronous, interactive, Web-based, e-learning training program (E-SOLAS) underpinned by behavior and learning theories was developed.

    Objective: This study investigated the effect of the E-SOLAS training program on relevant outcomes of effective training and implementation.

    Methods: Thirteen physiotherapists from across Ireland were trained via E-SOLAS by using mixed methods, and seven physiotherapists progressed to implementation of the 6-week group-based SOLAS intervention. The effectiveness of E-SOLAS was evaluated using the Kirkpatrick model at the levels of reaction (physiotherapist engagement and satisfaction with E-SOLAS training methods and content), learning (pre- to posttraining changes in physiotherapists’ confidence and knowledge in delivering SOLAS content and self-determination theory-based communication strategies, administered via a SurveyMonkey questionnaire), and behavior (fidelity to delivery of SOLAS content using physiotherapist-completed weekly checklists). During implementation, five physiotherapists audio recorded delivery of one class, and the communication between physiotherapists and clients was assessed using the Health Care Climate Questionnaire (HCCQ), the Controlling Coach Behaviour Scale (CCBS), and an intervention-specific measure (ISM; 7-point Likert scale). A range of implementation outcomes were evaluated during training and delivery (ie, acceptability, appropriateness, feasibility, fidelity, and sustainability of E-SOLAS) using a posttraining feedback questionnaire and individual semistructured telephone interviews.

    Results: With regard to their reaction, physiotherapists (n=13) were very satisfied with E-SOLAS posttraining (median 5.0; interquartile range 1.0; min-max 4.0-5.0) and completed training within 3-4 weeks. With regard to learning, there were significant increases in physiotherapists’ confidence and knowledge in delivery of all SOLAS intervention components (P<.05). Physiotherapists’ confidence in 7 of 10 self-determination theory-based communication strategies increased (P<.05), whereas physiotherapists’ knowledge of self-determination theory-based strategies remained high posttraining (P>.05). In terms of behavior, physiotherapists delivered SOLAS in a needs supportive manner (HCCQ: median 5.2, interquartile range 1.3, min-max 3.7-5.8; CCBS: median 6.6, interquartile range 1.0, min-max 5.6-7.0; ISM: median 4.5, interquartile range 1.2, min-max 2.8-4.8). Fidelity scores were high for SOLAS content delivery (total %mean fidelity score 93.5%; SD 4.9%). The posttraining questionnaire and postdelivery qualitative interviews showed that physiotherapists found E-SOLAS acceptable, appropriate, feasible, and sustainable within primary care services to support the implementation of the SOLAS intervention.

    Conclusions: This study provides preliminary evidence of the effectiveness, acceptability, and feasibility of an e-learning program to train physiotherapists to deliver a group-based self-management complex intervention in primary care settings, which is equivalent to face-to-face training outcomes and would support inclusion of physiotherapists in a definitive trial of SOLAS.

    J Med Internet Res 2019;21(3):e11123

    doi:10.2196/11123

    KEYWORDS



    Introduction

    International clinical guidelines for osteoarthritis and low-back pain endorse self-management, exercise, and physical activity as key components of health care interventions [1-4], but the evidence for their effectiveness is weak and of low quality [5-7]. The Self-management of Osteoarthritis and Low back pain through Activity and Skills (SOLAS) intervention is an evidence-supported group treatment approach developed through intervention mapping [8], which is a logical six-step process for the development and evaluation of theory-driven and evidence-based interventions that takes into account stakeholder needs and the practicalities of implementation [9]. SOLAS was evaluated for its acceptability and preliminary effects in comparison with individual physiotherapy in a feasibility trial (trial registration: ISRCTN49875385) set in Dublin, Ireland, between September 2014 and June 2016 [10]. Intervention physiotherapists who participated in the trial were trained using brief interactive lectures, videos, role play, and practical skills to deliver the SOLAS intervention using communication skills underpinned by self-determination theory. This theory proposes that people have basic psychological needs for autonomy, competence, and relatedness, which if met, for example, by the needs supportive communication style of a health care practitioner (HCP), will increase an individual’s autonomous motivation and engagement in health behaviors such as self-management [11]. The Medical Research Council guidelines recommend that complex behavior-change programs train their intervention deliverers to ensure implementation with high fidelity [12]. Hence, the Kirkpatrick model was used to evaluate training at the levels of Reaction, Learning, and Behavior [13], which showed that physiotherapists were satisfied with face-to-face training and their confidence in the self-determination theory-based communication strategies. Knowledge of the intervention content significantly increased, and the physiotherapists delivered SOLAS in a needs supportive manner with high fidelity to the intervention content [14,15]. Upscaling to a definitive national trial would render the face-to-face training impractical for physiotherapists due to significant time, travel, and costs constraints [16]. Therefore, we subsequently developed an asynchronous, interactive, Web-based, e-learning training program for SOLAS (E-SOLAS) to prepare physiotherapists to deliver the SOLAS intervention. If successful, the program would reduce the time needed to move to a definitive trial. Furthermore, E-SOLAS has the potential to increase the competencies of physiotherapists with regard to self-management behavior-change skills in line with the shared strategic priority of Ireland’s public health service and higher education institutions to train and prepare future health care graduates with the skills necessary to support lifestyle behavior change in their patients [17,18], making the intervention more accessible to physiotherapists for long-term sustainability.

    Despite the increased availability of e-learning training for HCPs internationally, there is limited formal evaluation of such training programs. Current evidence, which predominantly involves undergraduate HCP students [19,20], suggests that e-learning shows similar effectiveness to traditional methods for knowledge acquisition [21] and user satisfaction [22], but further research regarding the effectiveness of e-learning on HCP behavior change and the translation of learning to clinical practice has been advocated [23]. Hence, we evaluated E-SOLAS in the same way as our face-to-face training. In addition, a range of World Health Organization–recommended implementation outcomes were included for evaluation, including the acceptability, appropriateness, feasibility, fidelity, and sustainability of E-SOLAS, in a range of primary care physiotherapist settings across Ireland [24,25] in order to understand the contextual elements of e-learning [16].

    The study objectives were to evaluate the effect of the E-SOLAS training program on physiotherapists’ reaction, learning, and delivery of the SOLAS intervention as intended and to assess the acceptability, appropriateness, feasibility, fidelity, and sustainability of E-SOLAS to aid the implementation of the SOLAS intervention in primary care settings.


    Methods

    Study Design and Research Ethics

    This was a single-group, pre-post study. Ethical approval was granted by the UCD Human Subject (Sciences) Ethics Committee in two phases: in Phase 1, for the E-SOLAS training program (September 30, 2016; LS-E-16-121-Hurley) and in Phase 2, for implementation of the SOLAS intervention (December 21, 2016; LS-16-97-Hurley). The study was also approved for Phase 1 (November 17, 2016) and Phase 2 (January 17, 2017) by the Health Service Executive Primary Care Research Committee.

    Participants and Procedure

    Physiotherapy managers from 10 primary care areas across Ireland who had not participated in the SOLAS feasibility trial were sent a study information leaflet for screening based on their service facilities and staffing capabilities. Seven physiotherapy managers fulfilled the criteria for inclusion, provided letters of support, and nominated two staff members to undertake E-SOLAS training. Nominated physiotherapist staff were sent the study information leaflet and consent form. Consenting participants were required to possess a device that could connect to the internet and were given password-protected access to the social learning platform Curatr [26] that hosted the E-SOLAS training program. Participants were encouraged to complete the training over a 4-week period by working at their own pace and at times that were convenient for them. During training, they had access to ongoing technical support from the research team and were requested to keep a log of the time spent on each aspect of training.

    At the end of the training period, participants were invited to set up and deliver the SOLAS intervention according to the treatment protocol [10] in each of their primary care areas. Physiotherapists had ongoing access to E-SOLAS during implementation and were provided with any additional intervention materials required to deliver the intervention by the research team (ie, intervention PowerPoint [Microsoft Corp, Redmond, WA] slide deck on a universal serial bus, pedometers, and relaxation CDs for each client). Following completion of the 6-week delivery phase, each physiotherapist was invited to participate in an individual semistructured telephone interview to explore their views of E-SOLAS as a tool to support implementation of the intervention.

    E-SOLAS Training Program

    E-SOLAS is a Web-based e-learning training program designed to train physiotherapists to deliver a group-based education and exercise intervention for patients with osteoarthritis and chronic low-back pain. The content is based on the face-to-face training program developed for the SOLAS feasibility trial [10,15]. The E-SOLAS program is hosted on Curatr [26], an online social learning platform that creates a collaborative learning environment and uses gamification principles. The development process for the E-SOLAS program is outlined in Multimedia Appendix 1 [27-31].

    E-SOLAS Program Structure and Content

    The E-SOLAS program contained six hierarchical linear levels, whereby the user was required to finish each level before progressing to the next level (Multimedia Appendix 2, Figure 1).

    Briefly, the program begins in Level 1 with an overview of the training program and the SOLAS intervention. Level 2 describes the education content for each week of the SOLAS intervention (eg, the key learning points and the materials required; Figure 2). At Level 3, the self-determination theory-based communication strategies that physiotherapists use as part of the intervention are introduced (Figure 3), and in Level 4, they are given the opportunity to role play these strategies. Level 5 highlights the exercises and their mode of delivery, and finally, Level 6 concludes the program by highlighting the next steps for intervention delivery.

    Figure 1. E-SOLAS home page screenshot. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.
    View this figure
    Figure 2. E-SOLAS program content screenshot. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.
    View this figure
    Figure 3. E-SOLAS theory screenshot. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.
    View this figure
    Figure 4. E-SOLAS video-based program activity screenshot. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.
    View this figure

    Throughout each level, there are lectures with voice-overs, video examples of good and poor practice (Figure 4), videos from the research team, and a peer role model explaining certain elements of the intervention; short “in level” activities and self-reflection opportunities; and end-of-level “gate” assessments with varying levels and modes of feedback depending on the activity.

    Outcome Measures

    The effectiveness of the E-SOLAS training program was assessed using the Kirkpatrick model of evaluation at the levels of reaction, learning, and behavior [13]. Furthermore, a range of implementation outcomes were evaluated during the training and intervention-delivery phases. The measurement tools used to assess learning and implementation outcomes are described in detail in Multimedia Appendix 3 and briefly outlined below.

    Training Outcomes
    Reaction

    To measure physiotherapists’ reaction to E-SOLAS, a researcher-devised feedback measure was developed by adapting the face-to-face training feedback measure [15] and incorporating factors related to the evaluation of technology enhanced learning [25]. It was administered following posttraining assessment via SurveyMonkey and included items related to participant satisfaction, engagement, accessibility, and quality of E-SOLAS as well as several implementation outcomes as detailed below (Multimedia Appendix 4). Physiotherapists’ engagement with E-SOLAS was further evaluated using Curatr analytics [26] and a self-reported activity log completed by each physiotherapist during training.

    Learning

    Learning was assessed by evaluating physiotherapists’ perceptions of self-reported knowledge and confidence pre- and posttraining (Multimedia Appendix 5) and their use of skills during training by using a range of measures.

    Behavior

    Physiotherapists’ behavior was assessed during delivery of the SOLAS intervention to evaluate fidelity to the intervention content and self-determination theory-based communication strategies using previously validated checklists [14] and audio recordings [15]. In line with fidelity guidelines [32], each audio recording was coded by one blinded expert rater (AK) to assess physiotherapists’ communication style [33], and three audio recordings were coded by a second expert rater (JM). The Health Care Climate Questionnaire (HCCQ) [34] was the primary measure to assess provider delivery of the self-determination theory-based communication style, with an adapted version of the Controlling Coach Behaviour Scale (CCBS) [35] and an intervention-specific SOLAS scale used as secondary measures [15].

    Implementation Outcomes

    Implementation outcomes were measured through specific items on the feedback measure related to the acceptability, appropriateness, feasibility, and sustainability of E-SOLAS (Multimedia Appendix 4) and an individual semistructured telephone interview of the physiotherapists conducted by an experienced qualitative researcher (SG) within 2 weeks of completing group class delivery. A topic guide was developed for the participant interviews with specific questions and probes related to their views of E-SOLAS as a model of training in order to support physiotherapists in delivering the SOLAS intervention in primary care settings. All interviews were audio recorded.

    Data Analysis

    Data from all outcome measures were analyzed using Excel (version 14.2.3, Microsoft Corp) and a statistical software package (SPSS Statistics, version 20, IBM Corp, Armonk, NY) following checks for errors in data entry.

    Training Outcomes
    Reaction

    In order to assess physiotherapists’ views of their satisfaction, accessibility, and quality of the E-SOLAS program and their engagement with the e-learning training, descriptive statistics were used to analyze quantitative data, and thematic analysis was used to analyze free-text answers.

    Learning
    Analytical Methods

    Descriptive statistics were used to calculate scores pre- and posttraining for overall confidence in delivering SOLAS content, the specific SOLAS intervention components, and the use of each self-determination theory-based communication strategy. Differences between pre- and posttraining were calculated using the Wilcoxon signed-rank tests and adjusted for multiplicity using Bonferroni corrections (0.05/n tests).

    Knowledge

    Descriptive statistics were used to calculate the level of SOLAS intervention knowledge, and pre- and posttraining differences were calculated using the Wilcoxon signed-rank tests. Following discussion between raters, there was excellent agreement (100%) in the coding of physiotherapists’ narrative case studies. The number of self-determination theory-based communication strategies used by each physiotherapists and the percentage of physiotherapists who used each strategy was calculated, with differences in the rate of use of all strategies and each strategy pre- and posttraining determined using McNemar tests. All results were adjusted for multiplicity using Bonferroni corrections.

    Skills

    Each role-play audio recording was rated for the use of self-determination theory-based communications strategies on a 7-point Likert scale ranging from “1 - not at all well” to “7 - very well,” with values at or above the mid-point of the Likert scale (4/7) defined as demonstrating skills that could be considered acceptable in terms of competence [15].

    Behavior

    The mean fidelity levels to SOLAS intervention content and fidelity levels according to physiotherapist, site, session, and session category were obtained by calculating total actual scores as a percentage of the total possible score using checklists. Fidelity of duration was established by calculating the difference between the actual and the intended session durations using a one-sample Wilcoxon test. Levels of fidelity were interpreted as previously reported in the literature [36]. A review of the raters’ scores for the audio recordings of physiotherapists’ delivery of SOLAS session 4 demonstrated excellent agreement (90%). To establish physiotherapists’ competence in the self-determination theory-based communication style, a median result for each of the three outcome measures was calculated separately. For the SOLAS scale, a median score per construct subsection (eg, autonomy), subcomponent strategy (eg, positive feedback), and class component (eg, education) was also calculated.

    Implementation Outcomes

    Descriptive statistics were used to analyze quantitative data related to physiotherapists’ views of the acceptability, appropriateness, feasibility, and sustainability of E-SOLAS. Qualitative data from the physiotherapists’ interviews were transcribed verbatim and analyzed using inductive thematic analysis [37]. A coding frame was developed from a review of provisional themes, which were then reexamined and refined (DMcA). The reliability of the identified themes was established by a second researcher (DAH) who independently coded a random sample of 25% of each dataset using the coding frame, with 70% agreement taken as the minimum cut-off rate [10]. The level of agreement between raters was 85%.


    Results

    Principal Findings

    Thirteen physiotherapists from seven primary care areas completed the E-SOLAS training, of which 12 were invited to participate in the implementation study (ie, delivery of the SOLAS intervention). Nine physiotherapists consented to participate, and seven progressed to deliver SOLAS. The profile of physiotherapists in each study phase is provided in Table 1, and the flow of participants through Phase 2 is outlined in Figure 5. The training and delivery groups were comparable for the majority of descriptive variables, apart from the median years qualified, which was lower in the delivery group.

    Training Outcomes

    Reaction

    Physiotherapists (n=13) were very satisfied with E-SOLAS training posttraining and found it enjoyable and engaging, with all participants completing the program within the 4 weeks available (Multimedia Appendix 4). Physiotherapists reported that they spent a mean of 9.1 (SD 3.3) hours (min-max 4.1-16.1) over 16.3 (SD 6.0) days to complete E-SOLAS, which was not statistically different from the duration of training recorded by Curatr analytics (mean difference –1.69; SD 4.37; t=–1.397; df=12; P=.19; Multimedia Appendix 6). All physiotherapists successfully completed all-level gate assessments and the required three uploads and made at least one online posting to the group discussion. The majority of physiotherapists reported completing E-SOLAS outside work hours and spent 1-2 hours at any one time on training. The most commonly cited positive features of E-SOLAS were the range of brief video clips (46.2%; n=6) and focus on communication skills and client motivation (23.1%; n=3). Nine of the 13 participants experienced some difficulties during training; the most common difficulty was related to accessing online materials (46.2%, n=6), completing gate assessments (38.5%, n=5), and computer access at work (30.8%, n=4). Although the median ratings for working independently and not having access to other therapists were very positive, four physiotherapists required support from the University College Dublin team during training for accessing resources (n=3), logging into E-SOLAS via work email (n=2), or uploading audio files (n=1). Nonetheless, the majority of physiotherapists highly rated the quality of the training program and format.

    Table 1. Baseline characteristics of physiotherapists.
    View this table
    Figure 5. Participant flow through the study. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills; ID: participant identification number.
    View this figure
    Learning
    Knowledge

    Physiotherapists used all nine self-determination theory-based communication strategies in their responses pretraining; the most commonly used strategies were collaborative goal setting and action planning and building relationships, with no significant change in the rate of use of individual strategies posttraining (Table 2). Knowledge of the SOLAS intervention content and structure improved overall as well as in nine of the 10 intervention components. The use of pain modalities significantly increased posttraining, of which knowledge of content, structure, and group-based exercise programs remained significant following Bonferroni corrections (Table 3).

    Confidence

    Physiotherapists’ confidence significantly increased posttraining overall and for 7 of the 10 individual self-determination theory strategies; set clear expectations and provide direction remained significant after Bonferroni correction (Table 2).

    Similarly, physiotherapists’ confidence in delivery of the SOLAS content overall and all 10 intervention components significantly increased posttraining; five components remained significant after Bonferroni correction (Table 3).

    Table 2. Change in physiotherapists’ confidence and knowledge of self-determination theory-based communication strategies.
    View this table
    Table 3. Change in physiotherapists’ confidence and knowledge of the Self-management of Osteoarthritis and Low back pain through Activity and Skills intervention content.
    View this table
    Skills

    The majority of physiotherapists demonstrated acceptable use of the self-determination theory skill scores during training (median 5.0; interquartile range 1.3; min-max 2.0-6.0), with only two physiotherapists scoring <4.

    Behavior

    Of the six primary care sites that agreed to implement the intervention, five completed delivery and one site ceased delivery after Session 3 due to poor client attendance. Physiotherapists delivered SOLAS to a median of 4.0 (interquartile range 4; range 3-8) participants per class. The total mean %fidelity score (93.5%; SD 4.9%) and the overall fidelity scores were high (≥80%) (Multimedia Appendix 7). The difference between the actual and intended duration of all sessions was not statistically significant, apart from the education component of Session 1, which was significantly longer than the protocol (P=.03, Z=–2.23).

    Physiotherapists delivered SOLAS in a needs supportive manner consistent with a self-determination theory-based communication style (HCCQ: median 5.2, interquartile range 1.3, min-max 3.7-5.8; CCBS: median 6.6, interquartile range 1.0, min-max 5.6-7.0; Table 4). The SOLAS scale results demonstrated that physiotherapists implemented the intervention overall with acceptable competence (median 4.5, interquartile range 1.2, min-max 2.8-4.8; Table 4). The median scores of only 2 of the 15 self-determination theory strategies were delivered below the competence level during both the education and exercise components of the intervention (ie, use support and encouragement rather than pressurising behaviors and acknowledge patient’s feelings and perspectives).

    Implementation Outcomes

    Posttraining Feedback Questionnaire

    The median scores for physiotherapists’ ratings of the acceptability, appropriateness, and sustainability of E-SOLAS training to support delivery of the SOLAS intervention were high (Multimedia Appendix 4). All physiotherapists reported that E-SOLAS could be used as a training method in primary care, with 100% of respondents (n=13) recommending it to other primary care physiotherapists and the majority expressing a preference for e-learning alone (69.2%, n=9) over blended learning (30.8%, n=4).

    Postdelivery Qualitative Interviews

    Five of the seven physiotherapists who delivered the SOLAS intervention were interviewed within 2 weeks of program completion. Ten themes were identified from the analysis of participant interview data (Multimedia Appendix 8).

    Table 4. Physiotherapists’ use of the self-determination theory-based communication strategies during implementation of Session 4 of the Self-management of Osteoarthritis and Low back pain through Activity and Skills.
    View this table
    Acceptability of E-SOLAS

    Physiotherapists reported that they had a very positive experience with E-SOLAS training and felt that it was an acceptable and valuable method of training. A number of physiotherapists emphasized the convenience and flexibility of e-learning as a method of training as compared to face-to-face training. The format of training with gate-level assessments and the resource materials contained in E-SOLAS were also viewed positively.

    Appropriateness of E-SOLAS

    All five physiotherapists were positive about the appropriateness of the E-SOLAS content and resources in meeting their practical needs and the needs of their clients in preparing them to deliver SOLAS using needs supportive communication. One physiotherapist also reported gaining greater confidence in managing clients beyond the class setting, whereas another physiotherapist added that the e-learning format had the advantage of allowing her to reflect on learning new skills in relation to the autonomy-supporting style of delivery.

    Feasibility of E-SOLAS
    Demand

    Although all physiotherapists reported spending additional time reviewing the E-SOLAS content and resources in preparation for delivery of the intervention, they felt the additional time was important for the first delivery of any new program and would reduce with subsequent deliveries (Multimedia Appendix 9).

    Adaptation

    None of the physiotherapists reported deviating from the training specifications; however, all physiotherapists made recommendations for future adaptations to either the E-SOLAS content or training format. These included providing additional resources to guide physiotherapists in educating clients about the health risks associated with the overuse of pain medications, healthy eating guidelines, a wider range of exercise options, and the provision of outcome measures for clinicians to evaluate the intervention independently. Proposed adaptations to the training format included giving participants an estimate of the time required to complete each level and additional e-learning training and blended learning (ie, small-group face-to-face coaching alongside E-SOLAS) to support delivery of the self-determination theory-based communication strategies during goal setting and action-planning activities.

    Fidelity to E-SOLAS

    All physiotherapists aimed to deliver the intervention content and self-determination theory-based needs supportive communication with high fidelity.

    Sustainability of E-SOLAS

    Overall, physiotherapists were positive about the potential for integration of E-SOLAS into existing primary care settings to support the sustained use of the SOLAS intervention as a treatment and reported plans to continue implementation in their service area. One physiotherapist proposed training a designated clinician in each primary care area through face-to-face training, who would act as a peer mentor to support colleagues who completed E-SOLAS to specifically deliver the self-determination theory-based communication strategies.


    Discussion

    Overview

    The overall aim of this study was to develop and evaluate an e-learning training program to support physiotherapists to deliver the SOLAS intervention in a primary care setting. The effectiveness of E-SOLAS on physiotherapists’ knowledge, skills, and delivery of the SOLAS intervention was assessed alongside the acceptability and feasibility of the training program. Specifically, results indicated that physiotherapists’ knowledge and confidence increased from pretraining to posttraining assessment and physiotherapists’ behavior was positively influenced by E-SOLAS training, as the SOLAS intervention content and theory-based communication style were delivered with high fidelity. Finally, implementation outcomes posttraining and from the qualitative interviews were overtly positive with regard to the acceptability, appropriateness, feasibility, and potential for future integration of E-SOLAS into existing primary care health services.

    Effectiveness of the E-SOLAS Training Program

    Physiotherapists’ confidence in the SOLAS intervention content and self-determination theory-based communication style increased posttraining, which is important because it can indicate how likely a learner (in this case, physiotherapist) is to engage in the required behavior [38,39]. Knowledge also increased for the SOLAS intervention content, but there were limited changes in knowledge of the self-determination theory-based communication strategies. This may be explained by physiotherapists’ high pretraining knowledge levels, suggesting a ceiling effect, as they were highly experienced and the majority had undertaken communication-based training previously, thus limiting their potential for future improvement [40]. These findings mirror our previously published evaluation of SOLAS face-to-face training [15]. The majority of physiotherapists also demonstrated acceptable competence in relation to their skills; however, two physiotherapists were rated below the competence level. Review of the audio recordings revealed that one recording was very short (<2 minutes), and thus, it was difficult to assess it in a meaningful way. Although guidelines were provided on how to conduct the role play, no guidance was given on its duration. Despite the difficulty in prescribing a set amount of time, a minimum time period could have been set to ensure a meaningful assessment, which could be applied for future iterations of E-SOLAS.

    In terms of behavior, physiotherapists delivered the intervention as intended, adhering to the intervention content and delivery in a manner consistent with the self-determination theory-based communication style. The mean high fidelity to intervention content based on physiotherapists’ self-reported checklists was 93%, which aligns with the findings of the previous feasibility trial [14]. For assessment of the self-determination theory-based communication style, the physiotherapists’ scores on the two global measures aligned closely with face-to-face training. More specifically, for the HCCQ, the median score was 5.2 in this study and 5.3 (on a 7-point Likert scale) in the face-to-face training study [15]. Scores on the CCBS were consistent for both studies. However, an intervention-specific measure of needs support enables a more focused look at contextual elements [41]. Here, there was some divergence between e-training and face-to-face training, with median scores of 4.5 and 4.0 on a 7-point Likert scale, respectively, favoring E-SOLAS [15].

    In this study, two self-determination theory-based communication strategies (use support and encouragement rather than pressurising behaviours and acknowledge patients’ feelings and perspectives) were delivered with low competence across both the education and exercise components of the class, highlighting the need for further training or adaption to E-SOLAS to further support these strategies. Interestingly, the communication strategies related to goal setting, action planning, and problem solving were delivered to a higher level of competence than the face-to-face training [15]. This may have been due to the additional interactive elements added to E-SOLAS to address the concerns identified by the physiotherapists during the development phase. Furthermore, this improvement in goal setting-related strategies may have inadvertently reduced competence in the communication strategy use support and encouragement rather than pressurising behaviours, as emphasis was placed on physiotherapists being more directive with clients regarding goal setting in situations where clients were unable to articulate or formulate a goal themselves. Recent research has highlighted the difficulty in applying effective goal setting in clinical settings [42], and future training programs need to consider these strategies carefully. Overall, E-SOLAS training seems at least as effective as face-to-face training in developing physiotherapists’ knowledge, confidence, and ability to deliver the intervention as intended [22].

    Implementation Outcomes for the E-SOLAS Training Program

    Physiotherapists were very positive about E-SOLAS following training and delivery and believed it was an acceptable, appropriate, feasible, and sustainable method of training in primary care. Participants spent a mean of 9 hours completing the training over 16 days while working at their own pace and predominantly in their own time, which has clear advantages over the 12-hour face-to-face training time in addition to travel, cost, and time off work experienced by physiotherapists in our previous feasibility trial. E-SOLAS participants demonstrated high levels of engagement with training, including a 100% completion rate within the specified timeframe. This may reflect the physiotherapist-recognized importance of group-based self-management programs for busy primary care settings as well as the emphasis HCPs now place on a client-centered communication style and the acquisition of behavior-change skills [17]. Furthermore, these high levels of physiotherapists’ satisfaction and engagement could also reflect the systematic and inclusive process used to develop the E-SOLAS training program according to the recommendations of the Medical Research Council [12,43].

    In terms of feasibility, technical difficulties can sometimes hamper the success of e-learning with HCPs [44]. Six of the 13 physiotherapists reported difficulty accessing online materials. Therefore, it is important to ensure that technical support is in place to maintain user engagement. One of the main advantages of e-learning is flexibility and control of the time and location for program completion [45], as demonstrated in this study, wherein the majority of participants completed E-SOLAS outside work.

    Although physiotherapists were satisfied with the program overall, there were some adaptations suggested, including provision of further information to support the delivery of certain education components, inclusion of details of the estimated time to complete training, and the use of blended learning. These suggestions are in line with the general recommendations for e-learning programs should be tailored to HCPs’ particular knowledge and experience [46]. For example, in the context of E-SOLAS, one physiotherapist may want more information on pain medication, whereas another might like additional videos of communication strategies [45]. Such individualized learning pathways may lead to not only a more engaged learner with enhanced knowledge but also more effective delivery of the intervention.

    Despite the high rate of planned implementation of the SOLAS intervention posttraining, the program was fully delivered by six physiotherapists at five sites across four primary care areas. The main reasons for nonimplementation were beyond the scope of the study and were related to the nonavailability of staff. Of the five sites with full implementation, there was an equal mix of sole and shared delivery, in contrast to the previous feasibility trial where all physiotherapists delivered the intervention independently [15]. Physiotherapists who delivered the intervention implemented it with high fidelity, apart from the education component of Session 1, which is consistent with the findings of face-to-face training [14]. Although the qualitative interview findings did not suggest any significant barriers to future implementation by a sole practitioner following training, the suggestion of a local peer mentor and the development of blended learning may be warranted to overcome this potential obstacle.

    Strengths and Limitations

    The major strengths of this study are its focus on program development and evaluation within a group of experienced physiotherapists who received e-learning training while working within their primary care setting. Specifically, E-SOLAS was developed and underpinned by theory, with a clear rationale about how the intervention components were developed and adapted. The use of a formal evaluation model [13] allowed for a more comprehensive understanding of the effectiveness of E-SOLAS training, including the objective evaluation of physiotherapists’ behavior during training, which is frequently absent from assessments [23,47,48]. Furthermore, the application of the World Health Organization’s implementation outcomes using mixed methods enabled a comprehensive assessment of the feasibility of implementation of this e-learning training program and required adaptations to increase acceptability [24]. Finally, the assessment of fidelity of intervention delivery using validated measures following e-learning has been rarely reported in the literature and is one of the novel aspects of this study.

    A few limitations of this study should be acknowledged. Owing to the relatively small sample size, particularly for the delivery phase of the study, further investigation in a larger sample is warranted. Although a nonvalidated feedback measure was used to evaluate some training and implementation outcomes, its components were informed by a framework for the evaluation of technology-enhanced learning [25] and our face-to-face training feedback measure [15]. Although physiotherapists’ competence to deliver the SOLAS intervention was assessed posttraining, there was no pretraining assessment of their skills, which should be included in future studies [46]. Future studies should also incorporate some form of client measurement to more clearly understand the efficacy of this training approach. Self-report checklists were used to assess the fidelity to intervention content, which is less robust than other methods such as independently rated audio recordings [14]. Any future research evaluating a new program should apply robust fidelity-assessment methods to all parts of the intervention [36,49]. Finally, the role-play activities were an important part of E-SOLAS training; however, they were designed as one-on-one interactions (ie, between the physiotherapist and one client). Therefore, physiotherapists did not get an opportunity to practice their delivery of the intervention in a group setting prior to implementation. Future programs should try to ensure that all elements of the intervention are accurately reflected in the training program.

    Conclusions

    The comprehensive evaluation reported in this study provides preliminary evidence of the effectiveness, acceptability, and feasibility of an e-learning program to train physiotherapists to deliver a group-based self-management intervention in a primary care setting that is equivalent to face-to-face training. These findings will inform the development and implementation of a definitive trial and support its scalability to the wider primary care system.

    Acknowledgments

    The authors wish to thank the primary care physiotherapists and patients who participated in the study; William Fox and Isabelle Jeffares who supported the training phase of the study; and University College Dublin BSc Physiotherapy students Aine Moynan, Anna Scott, and Jessica Gayer for analysis of intervention content fidelity data. This article presents independent research funded by the Health Research Board (Knowledge Exchange and Dissemination Scheme 2015-1637).

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Description of the process used to develop the E-SOLAS program. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.

    PDF File (Adobe PDF File), 23KB

    Multimedia Appendix 2

    E-SOLAS training program structure and content. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.

    PDF File (Adobe PDF File), 40KB

    Multimedia Appendix 3

    Outcome measures for evaluation of E-SOLAS training and implementation of the SOLAS intervention. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.

    PDF File (Adobe PDF File), 21KB

    Multimedia Appendix 4

    Feedback measures used to record physiotherapists' reaction to E-SOLAS training and implementation outcomes posttraining. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.

    PDF File (Adobe PDF File), 57KB

    Multimedia Appendix 5

    Pre- and posttraining questionnaire for evaluation of physiotherapists' self-reported knowledge and perceived confidence of the SOLAS intervention and self-determination theory-based communication strategies. SOLAS: Self-management of Osteoarthritis and Low back pain through Activity and Skills.

    PDF File (Adobe PDF File), 55KB

    Multimedia Appendix 6

    Curatr analytics results.

    PDF File (Adobe PDF File), 23KB

    Multimedia Appendix 7

    Fidelity to delivery of SOLAS intervention content. SOLAS: Self-management of Osteoarthritis and Low back pain through Activity and Skills.

    PDF File (Adobe PDF File), 32KB

    Multimedia Appendix 8

    Acceptability and appropriateness of E-SOLAS training: Themes and theme examples from participant interviews. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.

    PDF File (Adobe PDF File), 53KB

    Multimedia Appendix 9

    Feasibility of E-SOLAS training: Themes and theme examples from participant interviews. E-SOLAS: E-learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills.

    PDF File (Adobe PDF File), 62KB

    References

    1. Fernandes L, Hagen KB, Bijlsma JW, Andreassen O, Christensen P, Conaghan PG, European League Against Rheumatism (EULAR). EULAR recommendations for the non-pharmacological core management of hip and knee osteoarthritis. Ann Rheum Dis 2013 Jul;72(7):1125-1135 [FREE Full text] [CrossRef] [Medline]
    2. Hochberg MC, Altman RD, April KT, Benkhalti M, Guyatt G, McGowan J, et al. American College of Rheumatology 2012 recommendations for the use of nonpharmacologic and pharmacologic therapies in osteoarthritis of the hand, hip, and knee. Arthritis Care Res (Hoboken) 2012 Apr;64(4):465-474. [Medline]
    3. National Institute for Health and Care Excellence. Osteoarthritis: care and management in adults   URL: https://www.nice.org.uk/guidance/cg177 [accessed 2018-05-23] [WebCite Cache]
    4. National Institute for Health and Care Excellence. Low back pain and sciatica in over 16s: assessment and management   URL: https://www.nice.org.uk/guidance/ng59 [accessed 2018-05-23] [WebCite Cache]
    5. Du S, Yuan C, Xiao X, Chu J, Qiu Y, Qian H. Self-management programs for chronic musculoskeletal pain conditions: a systematic review and meta-analysis. Patient Educ Couns 2011 Dec;85(3):e299-e310. [CrossRef] [Medline]
    6. Oliveira VC, Ferreira PH, Maher CG, Pinto RZ, Refshauge KM, Ferreira ML. Effectiveness of self-management of low back pain: systematic review with meta-analysis. Arthritis Care Res (Hoboken) 2012 Nov;64(11):1739-1748 [FREE Full text] [CrossRef] [Medline]
    7. Toomey E, Currie-Murphy L, Matthews J, Hurley DA. The effectiveness of physiotherapist-delivered group education and exercise interventions to promote self-management for people with osteoarthritis and chronic low back pain: a rapid review part I. Man Ther 2015 Apr;20(2):265-286. [CrossRef] [Medline]
    8. Hurley DA, Murphy LC, Hayes D, Hall AM, Toomey E, McDonough SM, et al. Using intervention mapping to develop a theory-driven, group-based complex intervention to support self-management of osteoarthritis and low back pain (SOLAS). Implement Sci 2016 Apr 26;11:56 [FREE Full text] [CrossRef] [Medline]
    9. Eldredge LKB, Parcel GS, Kok G, Gottlieb NH, Fernández ME. Planning Health Promotion Programs: An Intervention Mapping Approach. San Francisco: Jossey-bass; 2019.
    10. Hurley DA, Hall AM, Currie-Murphy L, Pincus T, Kamper S, Maher C, SOLAS Trial Team. Theory-driven group-based complex intervention to support self-management of osteoarthritis and low back pain in primary care physiotherapy: protocol for a cluster randomised controlled feasibility trial (SOLAS). BMJ Open 2016 Jan 21;6(1):e010728 [FREE Full text] [CrossRef] [Medline]
    11. Ng JYY, Ntoumanis N, Thøgersen-Ntoumani C, Deci EL, Ryan RM, Duda JL, et al. Self-Determination Theory Applied to Health Contexts: A Meta-Analysis. Perspect Psychol Sci 2012 Jul;7(4):325-340. [CrossRef] [Medline]
    12. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, et al. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655 [FREE Full text] [Medline]
    13. Kirkpatrick DKJ, Kirkpatrick JD. Evaluating Training Programs: The Four Levels (3rd Edition). San Francisco, CA: Berrett-koehler Publishers; 2019.
    14. Toomey E, Matthews J, Hurley DA. Using mixed methods to assess fidelity of delivery and its influencing factors in a complex self-management intervention for people with osteoarthritis and low back pain. BMJ Open 2017 Aug 04;7(8):e015452 [FREE Full text] [CrossRef] [Medline]
    15. Keogh A, Matthews J, Segurado R, Hurley DA. Feasibility of Training Physical Therapists to Deliver the Theory-Based Self-Management of Osteoarthritis and Low Back Pain Through Activity and Skills (SOLAS) Intervention Within a Trial. Phys Ther 2018 Feb 01;98(2):95-107. [CrossRef] [Medline]
    16. Richmond H, Hall AM, Hansen Z, Williamson E, Davies D, Lamb SE. Using mixed methods evaluation to assess the feasibility of online clinical training in evidence based interventions: a case study of cognitive behavioural treatment for low back pain. BMC Med Educ 2016 Jun 18;16:163 [FREE Full text] [CrossRef] [Medline]
    17. Health Service Executive. Making Every Contact Count: A Health Behaviour Change Framework and Implementation Plan for Health Professionals in the Irish Health Service. In: Health Service Executive. Dublin, Ireland: Health Service Executive; 2016.
    18. National Undergraduate Curriculum Working Group. National Undergraduate Curriculum for Chronic Disease Prevention and Management Part 1: Making Every Contact Count for Health Behaviour Change. In: Health Service Executive. Dublin, Ireland: Health Service Executive; 2017.
    19. Preston E, Ada L, Dean CM, Stanton R, Waddington G, Canning C. The Physiotherapy eSkills Training Online resource improves performance of practical skills: a controlled trial. BMC Med Educ 2012 Nov 26;12:119 [FREE Full text] [CrossRef] [Medline]
    20. Gardner P, Slater H, Jordan JE, Fary RE, Chua J, Briggs AM. Physiotherapy students' perspectives of online e-learning for interdisciplinary management of chronic health conditions: a qualitative study. BMC Med Educ 2016 Feb 16;16:62 [FREE Full text] [CrossRef] [Medline]
    21. Richmond H, Copsey B, Hall AM, Davies D, Lamb SE. A systematic review and meta-analysis of online versus alternative methods for training licensed health care professionals to deliver clinical interventions. BMC Med Educ 2017 Nov 23;17(1):227 [FREE Full text] [CrossRef] [Medline]
    22. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA 2008 Sep 10;300(10):1181-1196. [CrossRef] [Medline]
    23. Sinclair PM, Kable A, Levett-Jones T, Booth D. The effectiveness of Internet-based e-learning on clinician behaviour and patient outcomes: A systematic review. Int J Nurs Stud 2016 May;57:70-81. [CrossRef] [Medline]
    24. Peters DH, Adam T, Alonge O, Agyepong IA, Tran N. Implementation research: what it is and how to do it. BMJ 2013 Nov 20;347:f6753. [Medline]
    25. Cook DA, Ellaway RH. Evaluating technology-enhanced learning: A comprehensive framework. Med Teach 2015;37(10):961-970. [CrossRef] [Medline]
    26. Curatr. HT2 Labs   URL: https://www.ht2labs.com/curatr/ [accessed 2019-01-03] [WebCite Cache]
    27. Adams P. Exploring social constructivism: theories and practicalities. Education 3-13 2006 Oct;34(3):243-257. [CrossRef]
    28. Zimmerman BJ. Self-Regulated Learning and Academic Achievement: An Overview. Educational Psychologist 1990 Jan;25(1):3-17. [CrossRef]
    29. Ryan D, Deci E. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 2000 Jan;55(1):68-78. [Medline]
    30. Cane J, O'Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci 2012;7:37 [FREE Full text] [CrossRef] [Medline]
    31. Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med 2013 Aug;46(1):81-95. [CrossRef] [Medline]
    32. Borrelli B. The Assessment, Monitoring, and Enhancement of Treatment Fidelity In Public Health Clinical Trials. J Public Health Dent 2011;71(s1):S52-S63 [FREE Full text] [CrossRef] [Medline]
    33. Taylor IM, Ntoumanis N, Standage M. A self-determination theory approach to understanding the antecedents of teachers' motivational strategies in physical education. J Sport Exerc Psychol 2008 Feb;30(1):75-94. [Medline]
    34. Williams GC, Deci EL. Internalization of biopsychosocial values by medical students: a test of self-determination theory. J Pers Soc Psychol 1996 Apr;70(4):767-779. [Medline]
    35. Bartholomew KJ, Ntoumanis N, Thøgersen-Ntoumani C. The controlling interpersonal style in a coaching context: development and initial validation of a psychometric scale. J Sport Exerc Psychol 2010 Apr;32(2):193-216. [Medline]
    36. Toomey E, Matthews J, Guerin S, Hurley DA. Development of a Feasible Implementation Fidelity Protocol Within a Complex Physical Therapy-Led Self-Management Intervention. Phys Ther 2016 Aug;96(8):1287-1298. [CrossRef] [Medline]
    37. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006 Jan;3(2):77-101. [CrossRef]
    38. Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev 1977 Mar;84(2):191-215. [Medline]
    39. Ajzen I. The theory of planned behaviour: reactions and reflections. Psychol Health 2011 Sep;26(9):1113-1127. [CrossRef] [Medline]
    40. Su Y, Reeve J. A Meta-analysis of the Effectiveness of Intervention Programs Designed to Support Autonomy. Educ Psychol Rev 2010 Sep 25;23(1):159-188. [CrossRef]
    41. Fortier MS, Duda JL, Guerin E, Teixeira PJ. Promoting physical activity: development and testing of self-determination theory-based interventions. Int J Behav Nutr Phys Act 2012 Mar 02;9:20 [FREE Full text] [CrossRef] [Medline]
    42. Rose A, Rosewilliam S, Soundy A. Shared decision making within goal setting in rehabilitation settings: A systematic review. Patient Educ Couns 2017 Jan;100(1):65-75. [CrossRef] [Medline]
    43. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ 2015;350:h1258 [FREE Full text] [Medline]
    44. Eley R, Fallon T, Soar J, Buikstra E, Hegney D. The status of training and education in information and computer technology of Australian nurses: a national survey. J Clin Nurs 2008 Oct;17(20):2758-2767. [CrossRef] [Medline]
    45. Fontaine G, Cossette S, Heppell S, Boyer L, Mailhot T, Simard M, et al. Evaluation of a Web-Based E-Learning Platform for Brief Motivational Interviewing by Nurses in Cardiovascular Care: A Pilot Study. J Med Internet Res 2016 Aug 18;18(8):e224 [FREE Full text] [CrossRef] [Medline]
    46. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med 2010 May;85(5):909-922. [CrossRef] [Medline]
    47. Nicoll P, MacRury S, van Woerden HC, Smyth K. Evaluation of Technology-Enhanced Learning Programs for Health Care Professionals: Systematic Review. J Med Internet Res 2018 Apr 11;20(4):e131 [FREE Full text] [CrossRef] [Medline]
    48. Jacobs ZG, Elnicki DM, Perera S, Weiner DK. An E-learning Module on Chronic Low Back Pain in Older Adults: Effect on Medical Resident Attitudes, Confidence, Knowledge, and Clinical Skills. Pain Med 2018 Jun 1;19(6):1112-1120. [CrossRef] [Medline]
    49. Bellg AJ, Borrelli B, Resnick B, Hecht J, Minicucci DS, Ory M, Treatment Fidelity Workgroup of the NIH Behavior Change Consortium. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health Psychol 2004 Sep;23(5):443-451. [CrossRef] [Medline]


    Abbreviations

    CCBS: Controlling Coach Behaviour Scale
    E-SOLAS: E-Learning training program for Self-management of Osteoarthritis and Low back pain through Activity and Skills
    HCCQ: Health Care Climate Questionnaire
    HCP: health care practitioner
    ID: participant identification number
    SOLAS: Self-management of Osteoarthritis and Low back pain through Activity and Skills


    Edited by G Eysenbach; submitted 24.05.18; peer-reviewed by K Knittle, M El Tantawi; comments to author 30.09.18; revised version received 20.11.18; accepted 10.12.18; published 07.03.19.

    ©Deirdre A Hurley, Alison Keogh, Danielle Mc Ardle, Amanda M Hall, Helen Richmond, Suzanne Guerin, Tara Magdalinski, James Matthews. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 07.03.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.