JMIR Publications

Select Journals for Content Updates

When finished, please click submit.

This paper is in the following e-collection/theme issue:

    Original Paper

    Internal Versus External Motivation in Referral of Primary Care Patients with Depression to an Internet Support Group: Randomized Controlled Trial

    1Section of General Pediatrics Department of Pediatrics, University of Illinois at Chicago, Chicago, IL, United States

    2Dr. Bob LLC, Chicago, IL, United States

    3Division of Health Informatics and Implementation Science, University of Massachusetts Medical School, Worcester, MA, United States

    4eHealth QUERI, Bedford VA Medical Center, Bedford, MA, United States

    5Department of Finance and Business Management, Brooklyn College of the City University of New York, Brooklyn, NY, United States

    6Department of Psychiatry, The University of Chicago, Chicago, IL, United States

    7Division of General Internal Medicine, Department of Medicine, Johns Hopkins Medical Institutions, Baltimore, MD, United States

    *these authors contributed equally

    Corresponding Author:

    Benjamin W Van Voorhees, MD, M.Public Health

    Section of General Pediatrics Department of Pediatrics

    University of Illinois at Chicago

    840 S Wood St

    Suite 1437, MC 856

    Chicago, IL, 60612

    United States

    Phone: 1 3129960023

    Fax:1 3129960243

    Email:


    ABSTRACT

    Background: Depressive disorders and symptoms affect more than one-third of primary care patients, many of whom do not receive or do not complete treatment. Internet-based social support from peers could sustain depression treatment engagement and adherence. We do not know whether primary care patients will accept referral to such websites nor do we know which methods of referral would be most effective.

    Objective: We conducted a randomized clinical trial to determine whether (1) a simple generic referral card (control), (2) a patient-oriented brochure that provided examples of online postings and experience (internal motivation), or (3) a physician letter of recommendation (external motivation) would generate the greatest participation in a primary care Internet depression treatment support portal focused around an Internet support group (ISG).

    Methods: We used 3 offline methods to identify potential participants who had not used an ISG in the past 6 months. Eligibility was determined in part by a brief structured psychiatric interview based on the Patient Health Questionnaire-9 (PHQ-9). After consent and enrollment, participants were randomly assigned to 1 of 3 groups (control, internal motivation, or external motivation). We constructed a portal to connect primary care patients to both fact-based information and an established ISG (Psycho-Babble). The ISG allowed participants to view messages and then decide if they actually wished to register there. Participation in the portal and the ISG was assessed via automated activity tracking.

    Results: Fifty participants were assigned to the 3 groups: a motivation-neutral control group (n=18), an internal motivation group (n=19), and an external motivation group (n=13). Of these participants, 31 (62%) visited the portal; 27 (54%) visited the ISG itself. The internal motivation group showed significantly greater participation than the control group on several measures. The external motivation group spent significantly less time logged onto the portal than the control group. The internal motivation group showed significantly greater participation than the external motivation group on several measures.

    Conclusions: Referral of primary care patients with depressive disorders and symptoms to an ISG is feasible even if they have never previously used one. This may best be accomplished by enhancing their internal motivation.

    Trial Registration: Clinicaltrials.gov: NCT00886730; http://clinicaltrials.gov/show/NCT00886730 (Archived by WebCite at http://www.webcitation.org/6F4981fDN)

    J Med Internet Res 2013;15(3):e42)

    doi:10.2196/jmir.2197

    KEYWORDS



    Introduction

    Mood disorders have a lifetime prevalence of 20.8% [1-3]. Many Americans with a depressive disorder do not receive any treatment, and many of those who do receive treatment either do not receive high quality treatment or do not complete a full course of treatment [4]. Chronic care and collaborative care models have demonstrated benefit in improving process of care, symptoms, and functional outcomes [5-7]. A key component of these models appears to be support provided by case management, which may act by improving adherence or increasing patient activation [6,8]. However, these models are often expensive and cumbersome to implement [7]. As the number and quality of Internet models that provide self-directed psychotherapy [9], education [10], or social support [11] increases, so does the prospect for improving depression outcomes without costly person-to-person interventions or complex organizational changes [12]. Technology-based depression interventions have shown considerable promise in treating depression in select populations. However, the breadth of appeal of Internet-based treatment outside of highly structured surroundings, such as in school settings for children and adolescents, is not known [13-19].

    Internet sites offering peer-to-peer interaction may be particularly attractive to laypersons. In Australia, peer-to-peer and gaming models are widely used, including Reach Out! which provides psycho-education regarding common adolescent mental health problems [20]. Similarly, in the United States, social media is commonly used to address health concerns. Approximately 28% of Internet users are estimated to have contacted an Internet support group (ISG) for a medical condition or personal problem [21]. ISGs have several advantages, including transcending geographic barriers, facilitating disclosure (people will often tell a computer more than to another person), and increasing access to diverse sources of information. There is some evidence that use of ISGs for depression may be associated with both reductions in depressed mood and an increase in learning about depression [11,22]. Although we know little about how to engage primary care patients with Internet-based models, it is theorized that social engagement will be an essential component [17,23]. Unfortunately, many Internet-based interventions have high dropout rates, ranging up to 75% of all participants [24]. Participants’ longer and more consequential involvement with the Internet site and its direct personal relevance to the user predict greater behavior change in Internet-based programs [25,26]. Peer support and frequent updates may support participation in ISGs [27,28]. A prerequisite for further evaluations of the potential benefits of ISGs for primary care patients with depressive illness is developing an effective method of referral and ascertaining key predictors of participation. Self-determination theory provides a framework for evaluating possible referral methods. In this model, internal source motivations based upon preferences for autonomy, competence, and connection are superior to external source motivations based upon financial incentives or recommendation of an authority figure [2].

    The aim of this study was to determine the most feasible and effective methods to refer primary care patients with depressive disorders and symptoms to an ISG. We examined whether baseline attitudes based on the theories of motivational interviewing, planned behavior, and self-determination predicted Internet site participation [2,29], and we assessed whether a structured approach to goal setting would increase ISG participation [30]. We conducted a randomized controlled trial to compare 3 different methods of referring primary care patients with depressive disorders and symptoms to an ISG: emphasizing internal motivation (patient-oriented brochure), emphasizing external motivation (physician recommendation), and employing neither motivational strategy (referral card with Internet address). Interventions that require minimal physician involvement are particularly useful in actual practice settings. Because of this, the simple referral card is an important consideration. Our hypotheses were that (1) primary care referral (all methods) would be effective in engaging participants with the ISG (ie, >30% of participants would visit the ISG), (2) the physician letter recommendation (external motivation) group would show the greatest participation, (3) attitudes toward ISG participation would be important predictors of subsequent use, and (4) email reminders would increase participation.


    Methods

    Study Design

    This study included 9 key steps from initial recruitment to study completion. We identified potential participants using 3 methods: (1) reading a poster in the waiting room and self-referred, (2) completion of a statement of interest form in their physician’s office after discussion with their physician, or (3) completing a statement of interest at a public information table in the primary care office. In step 2, those recruited through reading a poster and self-referred or completing a statement of interest form after discussion with their physician provided consent to be called by study staff to learn more about the study (those who signed up at the public information table in the primary care office already knew about the study). Step 3 was a phone eligibility assessment. After providing consent for phone eligibility assessment, the study coordinator conducted a brief structured psychiatric interview based on the Patient Health Questionnaire-9 (PHQ-9) [31]. Following eligibility confirmation by both the principal investigator (internist-pediatrician) and co-investigator (psychiatrist), participants were offered enrollment in the study (step 4). There was approximately a 7-day delay between initial eligibility assessment and actual enrollment. At step 5, the study coordinator met the participants at or near the primary care office and conducted informed consent and enrollment. At this time, the participants also completed a baseline written questionnaire. In step 6 after consent and enrollment, participants were randomly assigned (using a sealed envelope with equal likelihood of assignment to all arms) to one of 3 groups: (1) referral card (neutral motivation group), (2) patient-oriented brochure providing examples of online postings and experience (internal motivation group), or (3) physician letter of recommendation (external motivation group) (see Figure 1). Participation on the Internet was assessed via automated activity tracking (step 7). Study staff contacted each participant 6 to 8 weeks after enrollment to evaluate depressed mood and any concerns about their website experiences (step 8). In the final step (step 9), each participant was asked to complete a written poststudy questionnaire. The study was approved by the University of Chicago Institutional Review Board. The trial was prospectively registered (ClinicalTrials.gov NCT00886730).

    Figure 1. CONSORT diagram.
    View this figure

    Inclusion and Exclusion Criteria

    All eligible participants met the following criteria: (1) a PHQ-9 score of 8 or above with either depressed mood or anhedonia and/or were considering treatment for depressed mood, (2) were accepting of at least one form of treatment for depression, (3) had not viewed or posted messages more than once in the past month on any ISG, (4) did not self-report being diagnosed with bipolar disorder by a health professional, (5) was 18 years of age or older, (6) attends a primary care clinic and had visited it in the past 6 months, and (7) had Internet access for the subsequent 4 weeks, had been on the Internet at least 3 times, and was able to use email. We excluded those considered to be at high risk for suicide attempts, which included those with past psychiatric hospitalizations, past suicide attempts, diagnosed bipolar disorder, or a score of greater than 1 on the PHQ-9 suicide assessment item (bothered by thoughts that he/she would be better off dead or of hurting him/herself in some way more than half the days in past 2 weeks), or who reported intent for self-harm as per assessment.

    Setting

    We recruited physicians at 6 primary care sites in urban and suburban areas of Chicago. These included a federally qualified health center (n=1), a private practice (n=1), university-affiliated off-site practices (n=2), a student health clinic (n=1), and an on-site academic resident practice (n=1). Physicians were recruited using lunchtime education programming.

    Adaptations Made After Study Initiation

    Several barriers to implementation were addressed after the study was fielded. During the course of the study, 2 practice sites closed and several physicians left the practices. After initial attempts using nurses to approach patients were unsuccessful, physicians were asked to approach appropriate patients directly to determine if they might be interested in the study. The original intention was to have the recommendation letter be from the participant’s personal physician. Instead, the letter was signed by the principal investigator (a primary care physician) because of time feasibility limitations.

    Randomization, Blinding, and Concealment

    We randomly assigned participants (using sealed envelopes) to 1 of the 3 groups described subsequently. A data manager who was not involved in the study execution prepared the sealed envelopes. Participants were aware of group assignments. The main outcome measures were collected in an automated fashion online. The database manager and the safety caller (described subsequently) were blinded to group assignments.

    Randomized Intervention Groups

    Neutral Motivation Orientation (Referral Card)

    Participants received a simple 3”×5” card with the name of the ISG and the following description: “Online support/coping group for people feeling blue, stressed, depressed, down, low, or sad” (see Multimedia Appendix 1 for full text).

    Internal Motivation Orientation (Patient-Centered Brochure)

    Participants received an 8”×11” handout that provided a more complete description of the ISG. The handout was based on a patient perspective with samples of Internet postings from current users. This card emphasized peer-to-peer support and did not mention health care organizations or health care provider endorsements. The information addressed potential barriers to participation: participants would not be identified, posting would not take much time, information from peers could be checked for accuracy with other peers and providers, and patients could learn how to tell their providers about their activities on the Internet (see Multimedia Appendix 1 for full text).

    External Motivation Orientation (Physician Recommendation)

    Participants received a medical recommendation in the form of a letter signed by the principal investigator, a primary care physician (see Multimedia Appendix 1 for full text).

    Email Reminder

    An email reminder was sent to all participants who did not visit the portal within 7 days of enrollment. Email reminders have been associated with higher levels of participation [27].

    Internet Portal Intervention

    We constructed a portal to connect primary care patients to both fact-based information and an established ISG. Two focus groups identified several key ideas with regard to the development of a primary care portal: (1) offer opportunities to learn from others, (2) offer opportunities to help others, (3) offer fact-based resources to evaluate accuracy of information, and (4) protect users from disturbing content, such as self-harm or salacious topics. To address these needs [32-34], we chose to build a portal (Psycho-Babble) that would: (1) offer access to fact-based information sites (eg, NIMH [35] and MOODGym [36]), (2) present aspects of the ISG that focus group participants suggested would be of greatest interest, and (3) offer access to the ISG. Participants could use the portal to visit the ISG and also the other websites. The ISG allowed participants to view messages and then decide if they actually wished to register there.

    Internet Support Group

    Psycho-Babble [37,38] is a mental health peer support group started in 1998 (see Figure 2). Online mental health groups can be classified as autonomous self-help groups or support groups led by mental health professionals. Psycho-Babble is a hybrid that combines the empowerment of self-help with the supportiveness of a milieu maintained by a mental health professional. The asynchronous online (message board) format makes the group more accessible and in some ways safer than groups that meet face-to-face [38]. The original forum continues to focus on biological treatments, and additional forums have been added for psychological treatments, complementary and alternative treatments, religious faith, social support, and discussion of the administration of the site. The portal through which participants reached Psycho-Babble highlighted 6 forums. The introduction in the main forum lists 3 other forums, and a link takes participants to a list of all 10. We expected participants to have been aware of the other Psycho-Babble forums. Higher user control has been associated with favorable perceptions by participants [39]. We believed Psycho-Babble would be a valuable ISG to use because of its well-developed participant base with history, loyalty, and repeat visits [40].

    Figure 2. Screenshot of Psycho-Babble home page.
    View this figure

    Physician and Nurse Training

    As part of this intervention, investigators met with each primary care provider and nurse at the participating primary care sites for 30 minutes during a lunchtime education program. This program included (1) explanation of the study design and the ISG and (2) summary of problems in depression care, self-management approaches currently used by individuals with depression, and the state of research on ISGs. A group of resident physicians received additional training on how to conduct randomized trials as an added inducement to participate in the study.

    Safety Monitoring

    A structured approach to managing participant safety was used. After eligibility assessment, all participants with elevated depressed mood received a recommendation for a medical evaluation. All those with self-harm ideation received a structured assessment to determine need for service. While enrolled in the study, the existing Psycho-Babble adverse event reporting mechanisms were available to participants. A safety call was made by a licensed clinical social worker 6 to 8 weeks after enrollment [37,38].

    Outcome Measures

    We evaluated use of the portal and the ISG by using data automatically logged by the portal and ISG servers (Table 1). We originally planned to assess outcomes across several categories: visits, time logged in, posts viewed, and posts posted. Few posts were posted; therefore, we added a broader measure of interaction, communications, which we use to refer to all submissions of data by participants. Participants submitted data not only for postings, but also to preview posts, to take the quiz required to register at the ISG, to enter a synchronous (real-time) chat room, to actually chat, and to contact the ISG administration.

    Portal Measures

    The portal server tracked visits and minutes logged in by each username. Participants were automatically logged out after 6 minutes. Time actually engaged with the portal may have been longer (eg, if they browsed the NIMH site for 30 minutes) or shorter (eg, if they stepped away from their computer after 3 minutes). If participants used the portal to visit the ISG, the time they spent at the ISG was included in the time they were logged in to the portal. However, if they visited the ISG directly, it was not. It was possible to access the ISG directly because it was open to the public and not restricted to study participants.

    Internet Support Group Measures

    At the ISG, viewing posts was unrestricted, but submitting data, including postings, was restricted to registered group members. The ISG server automatically tracked viewing of posts by participants by using cookies (information stored by a server on a user’s computer) which it set when participants visited the ISG from the portal. Because participants could visit the ISG directly or delete the cookies, viewing of posts (ie, ISG visits) by study participants may have been undercounted. The ISG server automatically tracked communications, including postings, by username.

    Independent Variables

    Sociodemographic Factors

    We obtained self-reported age (years), gender, race/ethnicity, and education level (eg, some high school or high school graduate).

    Mood

    For the participant’s dimensional measures of depressed mood, we used the 10-item Center for Epidemiologic Studies Depression Scale (CES-D) (Cronbach alpha =.87) [41] and the PHQ-9 [31] score (Cronbach alpha=.77).

    Self-efficacy

    Perceived control was assessed with the Mastery Scale [42]. The 7 items on the scale measure the extent to which participants see themselves as being in control of the forces that significantly affect their lives. Higher scores indicate greater self-efficacy (Cronbach alpha=.81).

    Loneliness

    The loneliness item of the CES-D was used to report loneliness. Respondents were asked the frequency of feeling lonely in the past week on a 4-point scale (0=less than a day, 1=1-2 days, 2=3-4 days, and 3=5-7 days).

    Self-determination Theory

    Self-determination theory posits that humans seek experiences in which they will develop autonomy, connection, and competence. Furthermore, the stronger the internal motivation (as opposed to external factors), the higher the quality and enduring the motivation [2]. Based on a focus group, we developed items to address attitudes in each of these domains and 1 additional one, concern for adverse experiences (see Multimedia Appendix 1). We measured 9 items based on a Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree) (Cronbach alpha=.69).

    Theory of Planned Behavior

    We revised the items from a previous questionnaire, which was modified based on the preventive health model and the theory of planned behavior [43,44] for the purpose of participation in our ISG. According to the theory of planned behavior, intention is the most proximal cognitive measure to actual behavior. Intention is directly influenced by attitudes and beliefs toward a behavior (eg, attitudes toward intervention), subjective norms (eg, concerns with regard to family, peer, or employer opinions), and perceived behavioral control (ie, controllability and self-efficacy) [29]. We measured items based on a Likert-type scale (1=strongly disagree to 5=strongly agree) for each domain (see Multimedia Appendix 1). The reliability (Cronbach alpha=.89) and predictive validity of the original instrument has previously been demonstrated in adolescents [45].

    Transtheoretical Model of Change

    Items were adapted from Miller and Rollnick’s 3-item assessment of motivation [46]. In terms of validity, this scale and many individual items predicted adherence to an Internet-based depression prevention intervention (modified form) [45]. On a scale from 1 to 10, participants rated the importance, their ability, and their readiness to overcome depressed mood over the next 2 months. Higher numerical values indicate higher levels for agreement with the item.

    Shared Decision Making

    We previously demonstrated improved outcomes with shared decision making for depression [47]. We asked participants to rate agreement with “I can talk with my providers in a way so my preferences for treatment are included” on a Likert-type scale (1=strongly disagree to 5=strongly agree).

    Analysis

    Descriptive statistics were calculated for the demographic variables. For between-group categorical comparisons, we used Pearson’s chi-square test or the Fisher exact test when there were <5 observations per cell. For continuous outcomes, we used analysis of variance (ANOVA) for between-group comparisons at the same time points. For continuous between-group data with skewed distribution, we used the Mann-Whitney test for 2-level comparisons. We used a similar analytical approach for comparisons between the 3 groups and also used the Kruskal-Wallis test for 3-level comparisons with a skewed distribution. We developed regression models for 3 main outcomes: (1) total time on-site, (2) number of posts viewed, and (3) number of posts attempted. Separate regression analyses were performed for the outcome variables. For total time on-site, the outcome had skewed data and there were values of zero, precluding a direct logarithmic transformation. Therefore, we added a value of 1 to all the variables so that they could be logarithmically transformed. Linear regression was then performed. For the count data (posts viewed and posts attempted), negative binomial regression was used. Because of the limited number of observations, we chose to develop only 2 models for each analysis. Model 1 was a univariate analysis. Model 2 adjusted for all items that were significant at P<.05 on Model 1 for each outcome. STATA 11 (StataCorp LP, College Station, TX, USA) was used to conduct the analyses. We also calculated effect sizes for between-group comparisons with statistical significance (Cohen’s d) [48].

    Sample Size Calculations and Interim Analyses

    A sample size of 225 was estimated to have a power of 0.8 to detect the difference between both the neutral group and the internal or external group (50%, 35%, and 10% participation). This sample size was not achieved because of an extensive delay in starting the study due to an extensive Institutional Review Board review (1 year). An interim analysis was conducted for purposes of safety review and end of grant funding. On the basis of the results of the interim analysis, which showed significant differences between randomization groups for the main study endpoints and significant predictors for measures of participation, the data monitoring and safety board (DSMB) and the investigators jointly agreed to end the trial.

    Table 1. Outcome variables.
    View this table

    Results

    Sample Characteristics

    A total of 50 participants were assigned to the 3 groups. At baseline, there were no significant differences between the 3 groups on any demographic characteristics (Table 2). With regard to the race/ethnicity of the sample, 47% were white (23/49), 41% were African American (20/49), 2% were Hispanic (1/49), 6% were Asian (3/49), and 4% were other (2/49). One participant did not report race/ethnicity. The mean age of participants was 37.49 years (SD 17.15). This was a moderately depressed cohort with PHQ-9 scores ranging between 7.22 to 10.24 for the 3 randomization groups with a mean for the entire sample of 9.18 (SD 4.64). With regard to education and socioeconomic status, 41% (20/48) were college graduates and the mean reported income was US $45,407 (SD 59,496). Fifty-one percent (25/49) reported discussing depression with their physician, 43% (21/49) reported being treated before for depression, and 31% (15/49) reported a family member had been depressed in the past. One participant did not provide any demographic data. Over the course of the study (November 2008 to June 2009), 128 individuals were referred by their physicians or answered advertisements posted in the clinics, of which 82 were successfully contacted and had eligibility assessed with 32 being excluded (see Multimedia Appendix 1). Reasons for exclusion included 10 who were eligible but did not enroll, 4 who were not interested, and 18 who did not meet eligibility criteria (8 with past psychiatric hospitalization, 8 with a self-harm attempt, and 2 with bipolar disorder). Of those who could be reached (82), only 4 reported no interest in the study. Primary care physicians recruiting for the study did not report many refusals for initial contact by study staff. Of those referred to the study staff for contact, 64% were contacted. Partial data were available for 2 participants, 1 who died of natural causes after enrollment (internal motivation group) and 1 participant enrolled in the external motivation group.

    Participation for Entire Cohort

    A total of 31 of 50 participants (62% of the entire study cohort) visited the portal or the ISG directly, whereas 27 of 50 (54%) registered on the ISG itself. Participants who visited the portal logged in for a mean of approximately 2 hours (118.5 minutes) each. Participants who visited the ISG viewed an average of 20.9 posts, communicated an average of 5.8 times, and posted an average of 0.4 posts. Table 3 shows the frequency distributions of the number of visits to the portal, time on the portal, and the number of posts viewed, communications, and posts posted. There was a skewed distribution spent on time on portal with 20 (40%) having no time, but 1 (2%) with 1301 minutes. Similarly, 4 (8%) participants viewed more than 46 posts, whereas 25 (50%) viewed no posts. In terms of communications, most (40/50, 80%) made no communications whereas a few (3/50, 6%) communicated more than 15 times. In terms of actual postings, 47 (94%) never posted, whereas 1 posted 1, 4, and 6 times. Most posts and replies were not replied to (new posts: 3 replied to, 6 not replied to).

    Participation by Randomization Group

    The internal motivation group had significantly greater participation for several participation measures than the neutral motivation group with more individual data submissions (mean 7.53 vs mean 0.39, P=.01, effect size [ES] 0.67, CI –0.02 to 1.32), were more likely to attempt to submit at least once (43% vs 6%, P=.02) and had significantly more posts attempted overall and by those who registered (mean 5.21 vs mean 0.17, P=.02, ES 0.62, CI –0.07 to 1.27 and mean 0.19 vs mean 0.01, P=.02, ES 0.55, CI –0.19 to 1.26, respectively). The neutral motivation group spent significantly more time logged onto the ISG as compared to the external motivation group (mean 9.27 vs mean 0.23, P=.04, ES 0.44, CI –0.29 to 1.15). There was only one 3-group comparison (see Multimedia Appendix 1) that demonstrated significance, which was the percent posting at least once (P=.02). The internal motivation group had significantly greater participation for several participation measures than the external motivation group (Tables 4 and 5). Specifically, members of the internal motivation group had more individual data submissions (mean 7.53 vs mean 0.54, P=.03, ES 0.60, CI –0.14 to 1.31), were more likely to submit at least once (42% vs 8%, P=.05), were more likely to have visited with the portal or ISG directly (79% vs 46%, P=.05) and were more likely to have attempted to post overall and by those who registered (mean 5.21 vs mean 0.23, P=.05, ES 0.56, CI –0.18 to 1.27 and mean 0.19 vs mean 0.01, P=.05, ES 0.55, CI –0.19 to 1.26, respectively). No other significant differences were observed.

    Email Reminders

    During the course of the study, 22 participants received an email reminder because they had not visited the portal within 7 days of enrollment. Of these 22 participants, 5 subsequently visited the portal; of those 5 participants, 3 did so within 2 days of receiving the email reminder.

    Predictors of Participation

    Time On-Site

    In the univariate analysis, demographic characteristics, severity of depressed mood, and self-efficacy did not predict minutes logged in (Table 6). One attitude item, “might read troubling comments,” was a statistically significant negative predictor (P=.03). Three items, “participating is important,” “want to do what physicians want me to do,” and “going to a depression Internet support website is easy,” were statistically significant (P=.049, P=.02, and P=.047, respectively), and another 3, “benefits outweigh difficulty,” “want to do what family thinks,” and “physicians think I should go” were not statistically significant (P=.06, P=.08, and P=.07, respectively). The motivation items and participation in decision making were not statistically significant. In the exploratory multivariate analysis of the significant univariate predictors, only the item of “might read troubling comments” remained statistically significant (P=.03).

    Posts Viewed

    Demographic characteristics and mood did not predict posts viewed (Table 7). Self-efficacy was not statistically significant (P=.09). No attitude items were statistically significant. Two intention items were statistically significant (“friends think I should go” P=.03, and “physicians think I should go” P=.01) and 1 was not statically significant (“want to do what friends think” P=.08). The motivation items and participation in decision making were not statistically significant. In the exploratory multivariate analysis of the significant univariate predictors, none remained statistically significant.

    Posts Attempted

    Demographic characteristics did not predict posts attempted (Table 8). Baseline CES-D, was not statically significant (P=.06). Self-efficacy was statistically significant (P=.04). Two intention items, “my family supports me going” and “physicians think I should go,” were statistically significant (P=.04 and P=.046, respectively), and another 2, “friends think I should go” and “want to do what physicians think,” were not statically significant (P=.06 and P=.06). Participation in decision making was not statistically significant. In the exploratory multivariate analysis of the significant univariate predictors, only self-efficacy remained statistically significant (P=.05).

    Safety and Unintended Events

    Fourteen of 49 participants received follow-up calls from the principal investigator related to concerns identified by the assessment caller (ie, depressed mood or self-harm ideation). One participant died of natural causes shortly after enrollment. No study participant reported adverse experiences while on the ISG and there were no self-harm events.

    Table 2. Demographics of study participants.
    View this table
    Table 3. Time on portal, posts viewed, and number of communications for entire cohort.
    View this table
    Table 4. Participation measures reporting only with P<.10.a
    View this table
    Table 5. Group comparisons of participation measures reporting only with P<.10.a
    View this table
    Table 6. Predictors of time on-site reporting only with P<.10.a
    View this table
    Table 7. Predictors of posts viewed reporting only with P<.10.a
    View this table
    Table 8. Predictors of posts attempted reporting only those with P<.10.a
    View this table

    Discussion

    This study suggests that low-cost and relatively low-intensity primary care referral methods are useful for engaging ISG-naive patients with depressed mood with a depression ISG. Our results support hypothesis 1 that referral of primary care patients to an Internet-based social support group portal was effective (>30% visited the site in all groups). Hypothesis 2 was not supported, whereby a patient-oriented brochure primarily focused on eliciting internal motivations demonstrated greater participation on multiple measures with moderate effect sizes than a generic medical letter of recommendation. Support for hypothesis 3 was attained with the findings that items from several models, but particularly the theory of planned behavior (in particular social norms), predicted participation. Furthermore, there was some evidence that email reminders to eligible patients were associated with the first visit to the portal (hypothesis 4).

    This is the first trial that we are aware of comparing low-intensity methods for referring primary care patients with depression to online social support or other online depression resources. The percentage of patients visiting the portal (79%) in the internal motivation (brochure) group was similar to those in other studies using minimal contact approaches in medical settings of brief advice (85%) and email reminder (77%) [49,50] versus participation in a mental health information website without any referral method in the general population (24%) [10]. We previously demonstrated that more intensive interventions (motivational interviewing versus brief advice) increases participation for Internet-based depression prevention interventions for adolescents, with similar moderate effect sizes [50]. Others have demonstrated that email-based reminders may increase participation [49]. Resnicow [51] demonstrated that health messaging using brochures that relate closely to the patients’ current concerns and preferred method of motivation may significantly increase prohealth behaviors such as diet change. The preparation of our brochure based on themes identified in 2 focus groups may have been particularly helpful. Similarly, relying on the brochure may have enabled our participants to develop higher quality intrinsic motivations as opposed to external ones. It is noteworthy that the greatest impact of the referral methods was on measures of posting, rather than simply viewing, which perhaps supports the premise that higher quality motivation is critical to enacting higher levels of engagement. Goal setting and having an established motivational structure to bring to the Internet experience has been noted to influence participation and could explain the added benefit of the intrinsic group (brochure) as well [30]. Similarly, perhaps the letter (external motivation) tended to crowd out intrinsic motivations in favor of external ones and serve as a negative incentive [52]. The finding that email reminders may increase participation is consistent with the findings of a prior systematic review [20]. However, it should be noted that the level of active involvement, such as posting, was very low (only 4 individuals posted).

    Attitudes and clinical factors, rather than demographic factors, provided a broader understanding of the process by which participants may choose to participate in our mental health ISG. Typically, factors that predict participation in a self-directed psychotherapy website include demographic factors (younger age, higher education, and greater illness severity), clinical factors (greater dysfunctional thinking and illness severity), health services factors (referral by a mental health professional), and attitudes (importance, self-efficacy, and perceived benefits) [50,53,54]. Although the attitudinal predictors of participation in an ISG are similar to those for Internet-based self-directed psychotherapy, social norms appear to be more important factors in predicting participation in an ISG [44,50,55-57]. Perhaps participants considered using an Internet-discussion group to be more of a social choice than a treatment decision or event, and when considering something unfamiliar they defer to the judgment of others. It is uncertain how much participation on an ISG could be as sensitive to comparisons of activity with other individuals such as that occurs with gaming on the Internet [28].

    These sites reflected the challenges of contemporary primary care, as 2 of the practice sites closed during the study and were relocated as a result of the financial crisis of 2008. In terms of internal validity, the participants were effectively randomized with no significant between-group differences and received 3 distinct interventions. Although small, the sample size demonstrated between-group differences and helped illuminate several preliminary associations. However, several key limitations with internal validity should be recognized: (1) the sample was quite small and the smallest group had only 13 participants (we only analyzed 12), (2) a letter from the patient’s own physician personal letter of recommendation might have performed better than a generic one, (3) the closure of 2 clinics during the study disrupted recruiting and in some cases delivery of the intervention elements, (4) delivery of emails to all groups and direct physician recruitment of nearly all participants may have resulted in reduced between-group differences in participation, and (5) poor engagement may simply have reflected lack of easy access to the Internet or experience with interfaces like those of the portal and ISG. The observation that the external motivation (recommendation) group assignment had a mean PHQ-9 of 7.22, which was below the eligibility criteria, suggests this group had disproportionate improvement in depressed mood before actual enrollment. Two forms of bias may explain this: (1) practice effect bias suggests that if the same test is administered within 2 weeks of time then practice effect bias is present (ie, a person remembers the questions and might try to change the answers), or (2) experimenter bias (participants may have guessed we were expecting to find decreased depression and answered to satisfy us). This could suggest that this group varied from the others in some unmeasured characteristic, such as motivation, but this seems unlikely to alter the result.

    In terms of external validity, we should note several concerns as well. Although it is possible that some participants were not ISG-naive, the exclusion of only those who had not visited a depression ISG within the past 6 months is unlikely to have resulted in recruitment of participants who were significantly more disposed to depression ISGs than the general population. Because we did not collect data on prior use of other depression ISGs, we cannot determine the extent to which prior use may have influenced participation in the ISG of this study. Similarly, the recruiting methods may have self-selected for more motivated individuals, a fact that may suggest caution in estimating future effectiveness in primary care. However, recruiting physicians did not report many patients refusing referrals to the study and 64% of those referred were contacted. Given that this was perhaps a somewhat motivated sample, just half of the participants actually registered on the ISG. Lack of Internet access in this relatively lower income population and a somewhat outdated interface may have reduced participation versus a population with greater Internet access and skill [58]. Similarly, the degree to which the website may not have met current expectations for frequently updated materials, such as the style of the site, may have affected participation [58,59].

    In conclusion, primary care patients may be amenable to referral to Internet-based social support and information portals, even if they have not utilized ISGs in the past. Relatively low-cost materials and minimal physician interaction may be required to accomplish this referral process. Motivational approaches focused on intrinsic factors may be superior to extrinsic ones. Referral to Internet-based social support using relatively low-cost methods, such as brochures, may offer a key method to moving toward patient-centered models, particularly for an illness in which treatment adherence and perhaps even outcomes are substantially influenced by prevailing social norms and attitudes. Clinicians may wish to consider referral to such reliable and reputable Internet-based social media as an early step to enable patients to explore with others treatment options or barriers to adherence. Health administrators and health services researchers should consider streamlined approaches to engaging patients with reliable and reputable Internet-based supportive social media. The next logical step would be a randomized clinical trial in a primary care setting to determine if referral to Internet-based social support for those with depressive illness improves quality of care, symptoms, and function.

    Acknowledgments

    Supported by NIMH R-34 R34 MH 073742-01 and NIMH K-08 MH 072918-01A2 (Dr Van Voorhees salary support during study).

    Conflicts of Interest

    Dr Van Voorhees operates a limited liability company, RISE Consulting, that provides know-how support for health technology companies. Dr Van Voorhees has served as a consultant for Prevail Health Solutions, Inc, Mevident Inc, Verimed Inc, and Social Kinetics, Inc. Dr Van Voorhees has also been a paid speaker/academic visitor at the University of Hong Kong, Chinese International School (Hong Kong), and Wuhan University. The University of Chicago has granted a no-cost license to Mevident, Inc to adapt the CATCH-IT intervention. Dr Van Voorhees has agreed to support the company working 5.5 days as consultant at $1000/day.

    Dr Hsiung is the single member of Dr Bob LLC, which owns and operates the Psycho-Babble Internet support group. The net revenue of Dr Bob LLC is less than $1000/year.

    Multimedia Appendix 1

    Group assignment samples and extended analysis of the tables embedded in the body of the manuscript.

    PDF File (Adobe PDF File), 119KB

    Multimedia Appendix 2

    CONSORT eHealth Checklist V1.6.2 [36].

    PDF File (Adobe PDF File), 1008KB

    References

    1. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 2005 Jun;62(6):593-602. [CrossRef] [Medline]
    2. Vansteenkiste M, Sheldon KM. There's nothing more practical than a good theory: integrating motivational interviewing and self-determination theory. Br J Clin Psychol 2006 Mar;45(Pt 1):63-82. [CrossRef] [Medline]
    3. Kessler RC, Nelson CB, McGonagle KA, Liu J, Swartz M, Blazer DG. Comorbidity of DSM-III-R major depressive disorder in the general population: results from the US National Comorbidity Survey. Br J Psychiatry Suppl 1996 Jun(30):17-30. [Medline]
    4. Kessler RC, Zhao S, Katz SJ, Kouzis AC, Frank RG, Edlund M, et al. Past-year use of outpatient services for psychiatric problems in the National Comorbidity Survey. Am J Psychiatry 1999 Jan;156(1):115-123. [Medline]
    5. Wells K, Sherbourne C, Schoenbaum M, Ettner S, Duan N, Miranda J, et al. Five-year impact of quality improvement for depression: results of a group-level randomized controlled trial. Arch Gen Psychiatry 2004 Apr;61(4):378-386. [CrossRef] [Medline]
    6. Gilbody S, Bower P, Fletcher J, Richards D, Sutton AJ. Collaborative care for depression: a cumulative meta-analysis and review of longer-term outcomes. Arch Intern Med 2006 Nov 27;166(21):2314-2321. [CrossRef] [Medline]
    7. Gilbody S, Bower P, Whitty P. Costs and consequences of enhanced primary care for depression: systematic review of randomised economic evaluations. Br J Psychiatry 2006 Oct;189:297-308 [FREE Full text] [CrossRef] [Medline]
    8. Gensichen J, Beyer M, Muth C, Gerlach FM, Von Korff M, Ormel J. Case management to improve major depression in primary health care: a systematic review. Psychol Med 2006 Jan;36(1):7-14. [CrossRef] [Medline]
    9. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther 2009;38(4):196-205. [CrossRef] [Medline]
    10. Santor DA, Poulin C, LeBlanc JC, Kusumakar V. Online health promotion, early identification of difficulties, and help seeking in young people. J Am Acad Child Adolesc Psychiatry 2007 Jan;46(1):50-59. [CrossRef] [Medline]
    11. Griffiths KM, Calear AL, Banfield M. Systematic review on Internet Support Groups (ISGs) and depression (1): Do ISGs reduce depressive symptoms? J Med Internet Res 2009;11(3):e40 [FREE Full text] [CrossRef] [Medline]
    12. Landback J, Prochaska M, Ellis J, Dmochowska K, Kuwabara SA, Gladstone T, et al. From prototype to product: development of a primary care/internet based depression prevention intervention for adolescents (CATCH-IT). Community Ment Health J 2009 Oct;45(5):349-354. [CrossRef] [Medline]
    13. Whittaker R. MEMO--a mobile phone depression prevention intervention for adolescents: development process and postprogram findings on acceptability from a randomized controlled trial. J Med Internet Res 2012;14(1).
    14. Ritterband LM, Ardalan K, Thorndike FP, Magee JC, Saylor DK, Cox DJ, et al. Real world use of an Internet intervention for pediatric encopresis. J Med Internet Res 2008;10(2):e16 [FREE Full text] [CrossRef] [Medline]
    15. Samoocha D, Bruinvels DJ, Elbers NA, Anema JR, van der Beek AJ. Effectiveness of web-based interventions on patient empowerment: a systematic review and meta-analysis. J Med Internet Res 2010;12(2):e23 [FREE Full text] [CrossRef] [Medline]
    16. van Straten A, Cuijpers P, Smits N. Effectiveness of a web-based self-help intervention for symptoms of depression, anxiety, and stress: randomized controlled trial. J Med Internet Res 2008;10(1):e7 [FREE Full text] [CrossRef] [Medline]
    17. Van Voorhees B, Watson N, Bridges JF, Fogel J, Galas J, Kramer C, et al. Development and pilot study of a marketing strategy for primary care/internet-based depression prevention intervention for adolescents (the CATCH-IT intervention). Prim Care Companion J Clin Psychiatry 2010;12(3) [FREE Full text] [CrossRef] [Medline]
    18. Van Voorhees BW, Mahoney N, Mazo R, Barrera AZ, Siemer CP, Gladstone TR, et al. Internet-based depression prevention over the life course: a call for behavioral vaccines. Psychiatr Clin North Am 2011 Mar;34(1):167-183 [FREE Full text] [CrossRef] [Medline]
    19. Cuijpers P, Donker T, van Straten A, Li J, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med 2010 Dec;40(12):1943-1957. [CrossRef] [Medline]
    20. Burns J, Morey C, Lagelée A, Mackenzie A, Nicholas J. Reach Out! Innovation in service delivery. Med J Aust 2007 Oct 1;187(7 Suppl):S31-S34. [Medline]
    21. Rice RE. Influences, usage, and outcomes of Internet health information searching: multivariate results from the Pew surveys. Int J Med Inform 2006 Jan;75(1):8-28. [CrossRef] [Medline]
    22. Griffiths KM, Calear AL, Banfield M, Tam A. Systematic review on Internet Support Groups (ISGs) and depression (2): What is known about depression ISGs? J Med Internet Res 2009;11(3):e41 [FREE Full text] [CrossRef] [Medline]
    23. Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res 2011;13(1):e30 [FREE Full text] [CrossRef] [Medline]
    24. Kaltenthaler E, Brazier J, De Nigris E, Tumur I, Ferriter M, Beverley C, et al. Computerised cognitive behaviour therapy for depression and anxiety update: a systematic review and economic evaluation. Health Technol Assess 2006 Sep;10(33):iii, xi-ixiv, 1 [FREE Full text] [Medline]
    25. Strecher VJ, Shiffman S, West R. Moderators and mediators of a web-based computer-tailored smoking cessation program among nicotine patch users. Nicotine Tob Res 2006 Dec;8 Suppl 1:S95-101. [Medline]
    26. Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ Res 2008 Jun;23(3):427-439 [FREE Full text] [CrossRef] [Medline]
    27. Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011;13(1):e2 [FREE Full text] [CrossRef] [Medline]
    28. Lin RJ, Zhu X. Leveraging social media for preventive care-A gamification system and insights. Stud Health Technol Inform 2012;180:838-842. [Medline]
    29. Ajzen I. The directive influence of attitudes on behavior. In: Gollwitzer PM, Bargh JA, editors. The Psychology of Action: Linking Cognition and Motivation to Behavior. New York: Guilford Press; 1996:385-403.
    30. Cugelman B, Thelwall M, Dawes P. Online interventions for social marketing health behavior change campaigns: a meta-analysis of psychological architectures and adherence factors. J Med Internet Res 2011;13(1):e17 [FREE Full text] [CrossRef] [Medline]
    31. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [Medline]
    32. Sturm R. Tracking changes in behavioral health services: how have carve-outs changed care? J Behav Health Serv Res 1999 Nov;26(4):360-371. [Medline]
    33. Sturm R. How expensive is unlimited mental health care coverage under managed care? JAMA 1997 Nov 12;278(18):1533-1537. [Medline]
    34. Shuman CR. Managed psychiatric care: a suburban medical department activity model. Mil Med 1996 Sep;161(9):557-561. [Medline]
    35. National Institute of Mental Health.   URL: http://www.nimh.nih.gov/index.shtml [accessed 2013-02-11] [WebCite Cache]
    36. Australian National University. The MoodGYM Training Program   URL: https://moodgym.anu.edu.au/welcome [accessed 2013-02-11] [WebCite Cache]
    37. Hsuing R. Psycho-Babble.   URL: http://www.dr-bob.org/babble/ [accessed 2013-02-11] [WebCite Cache]
    38. Hsiung R. The best of both worlds: An online self-help group hosted by a mental health professional. CyberPsychology & Behavior 2000 Dec 2000;3(6):935-950. [CrossRef]
    39. Crutzen R, Cyr D, de Vries NK. The role of user control in adherence to and knowledge gained from a website: randomized comparison between a tunneled version and a freedom-of-choice version. J Med Internet Res 2012;14(2):e45 [FREE Full text] [CrossRef] [Medline]
    40. Crutzen R, Cyr D, de Vries NK. Bringing loyalty to e-Health: theory validation using three internet-delivered interventions. J Med Internet Res 2011;13(3):e73 [FREE Full text] [CrossRef] [Medline]
    41. Radloff LS. The CES-D scale: a self-report depression scale for research in the general population. Applied Psychological Measures 1977;1:385-401.
    42. Pearlin LI, Lieberman MA, Menaghan EG, Mullan JT. The stress process. J Health Soc Behav 1981 Dec;22(4):337-356. [Medline]
    43. Myers RE, Wolf TA, McKee L, McGrory G, Burgh DY, Nelson G, et al. Factors associated with intention to undergo annual prostate cancer screening among African American men in Philadelphia. Cancer 1996 Aug 1;78(3):471-479. [CrossRef] [Medline]
    44. Van Voorhees BW, Fogel J, Houston TK, Cooper LA, Wang NY, Ford DE. Attitudes and illness factors associated with low perceived need for depression treatment among young adults. Soc Psychiatry Psychiatr Epidemiol 2006 Sep;41(9):746-754. [CrossRef] [Medline]
    45. Marko M, Fogel J, Mykerezi E, Van Voorhees BW. Adolescent Internet depression prevention: preferences for intervention and predictors of intentions and adherence. Journal of CyberTherapy and Rehabilitation 2010;3(1):9-30.
    46. Miller W, Rollnick S. Motivational Interviewing: Preparing People to Change Addictive Behaviors. New York: Guilford Press; 1991.
    47. Clever SL, Ford DE, Rubenstein LV, Rost KM, Meredith LS, Sherbourne CD, et al. Primary care patients' involvement in decision-making is associated with improvement in depression. Med Care 2006 May;44(5):398-405. [CrossRef] [Medline]
    48. Cohen J. Statistical Power Analysis for the Behavioral Sciences 2nd ed. Hillsdale, NJ: Erlbaum; 1988.
    49. Ritterband LM, Borowitz S, Cox DJ, Kovatchev B, Walker LS, Lucas V, et al. Using the internet to provide information prescriptions. Pediatrics 2005 Nov;116(5):e643-e647 [FREE Full text] [CrossRef] [Medline]
    50. Van Voorhees BW, Fogel J, Pomper BE, Marko M, Reid N, Watson N, et al. Adolescent Dose and Ratings of an Internet-Based Depression Prevention Program: A Randomized Trial of Primary Care Physician Brief Advice versus a Motivational Interview. J Cogn Behav Psychother 2009;9(1):1-19 [FREE Full text] [Medline]
    51. Resnicow K, Davis RE, Zhang G, Konkel J, Strecher VJ, Shaikh AR, et al. Tailoring a fruit and vegetable intervention on novel motivational constructs: results of a randomized study. Ann Behav Med 2008 Apr;35(2):159-169. [CrossRef] [Medline]
    52. Hoek W, Marko M, Fogel J, Schuurmans J, Gladstone T, Bradford N, et al. Randomized controlled trial of primary care physician motivational interviewing versus brief advice to engage adolescents with an Internet-based depression prevention intervention: 6-month outcomes and predictors of improvement. Transl Res 2011 Dec;158(6):315-325. [CrossRef] [Medline]
    53. Batterham PJ, Neil AL, Bennett K, Griffiths KM, Christensen H. Predictors of adherence among community users of a cognitive behavior therapy website. Patient Prefer Adherence 2008;2:97-105 [FREE Full text] [Medline]
    54. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res 2009;11(2):e13 [FREE Full text] [CrossRef] [Medline]
    55. Veroff J, Kulka RA, Douvan E. Mental Health in America: Patterns of Help Seeking from 1957-1976. New York: Basic Books; 1981.
    56. Van Voorhees BW, Fogel J, Houston TK, Cooper LA, Wang NY, Ford DE. Beliefs and attitudes associated with the intention to not accept the diagnosis of depression among young adults. Ann Fam Med 2005;3(1):38-46 [FREE Full text] [CrossRef] [Medline]
    57. Nutting P, Rost K, Dickinson M, Werner JJ, Dickinson P, Smith JL, et al. Barriers to initiating depression treatment in primary care practice. J Gen Intern Med 2002 Feb;17(2):103-111 [FREE Full text] [Medline]
    58. Schneider F, van Osch L, de Vries H. Identifying factors for optimal development of health-related websites: a delphi study among experts and potential future users. J Med Internet Res 2012;14(1):e18 [FREE Full text] [CrossRef] [Medline]
    59. Brouwer W, Oenema A, Crutzen R, de Nooijer J, de Vries NK, Brug J. An exploration of factors related to dissemination of and exposure to internet-delivered behavior change interventions aimed at adults: a Delphi study approach. J Med Internet Res 2008;10(2):e10 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    ANOVA: analysis of variance
    CES-D: Center for Epidemiologic Studies Depression Scale
    ES: effect size
    ISG: Internet support group
    PHQ-9: Patient Health Questionnaire-9


    Edited by G Eysenbach; submitted 01.06.12; peer-reviewed by C Barker, J Barnes; comments to author 25.08.12; revised version received 26.10.12; accepted 19.12.12; published 12.03.13

    ©Benjamin W Van Voorhees, Robert C Hsiung, Monika Marko-Holguin, Thomas K Houston, Joshua Fogel, Royce Lee, Daniel E Ford. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 12.03.2013.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.