Published on in Vol 20, No 5 (2018): May

Preprints (earlier versions) of this paper are available at, first published .
Developing a Mental Health eClinic to Improve Access to and Quality of Mental Health Care for Young People: Using Participatory Design as Research Methodologies

Developing a Mental Health eClinic to Improve Access to and Quality of Mental Health Care for Young People: Using Participatory Design as Research Methodologies

Developing a Mental Health eClinic to Improve Access to and Quality of Mental Health Care for Young People: Using Participatory Design as Research Methodologies

Original Paper

Corresponding Author:

Laura Ospina-Pinillos, MD

Brain and Mind Centre

The University of Sydney

94 Mallett Street, Camperdown

Sydney, 2050


Phone: 61 0286276946


Background: Each year, many young Australians aged between 16 and 25 years experience a mental health disorder, yet only a small proportion access services and even fewer receive timely and evidence-based treatments. Today, with ever-increasing access to the Internet and use of technology, the potential to provide all young people with access (24 hours a day, 7 days a week) to the support they require to improve their mental health and well-being is promising.

Objective: The aim of this study was to use participatory design (PD) as research methodologies with end users (young people aged between 16 and 25 years and youth health professionals) and our research team to develop the Mental Health eClinic (a Web-based mental health clinic) to improve timely access to, and better quality, mental health care for young people across Australia.

Methods: A research and development (R&D) cycle for the codesign and build of the Mental Health eClinic included several iterative PD phases: PD workshops; translation of knowledge and ideas generated during workshops to produce mockups of webpages either as hand-drawn sketches or as wireframes (simple layout of a webpage before visual design and content is added); rapid prototyping; and one-on-one consultations with end users to assess the usability of the alpha build of the Mental Health eClinic.

Results: Four PD workshops were held with 28 end users (young people n=18, youth health professionals n=10) and our research team (n=8). Each PD workshop was followed by a knowledge translation session. At the conclusion of this cycle, the alpha prototype was built, and one round of one-on-one end user consultation sessions was conducted (n=6; all new participants, young people n=4, youth health professionals n=2). The R&D cycle revealed the importance of five key components for the Mental Health eClinic: a home page with a visible triage system for those requiring urgent help; a comprehensive online physical and mental health assessment; a detailed dashboard of results; a booking and videoconferencing system to enable video visits; and the generation of a personalized well-being plan that includes links to evidence-based, and health professional–recommended, apps and etools.

Conclusions: The Mental Health eClinic provides health promotion, triage protocols, screening, assessment, a video visit system, the development of personalized well-being plans, and self-directed mental health support for young people. It presents a technologically advanced and clinically efficient system that can be adapted to suit a variety of settings in which there is an opportunity to connect with young people. This will enable all young people, and especially those currently not able or willing to connect with face-to-face services, to receive best practice clinical services by breaking down traditional barriers to care and making health care more personalized, accessible, affordable, and available.

J Med Internet Res 2018;20(5):e188




The Internet and emerging technologies have long been identified as having the potential to significantly expand the reach of quality mental health care by addressing geographical, economical, and human resource barriers [1,2]. Over the past decade, evidence suggests that the Internet is considered not only a major source of information about health (including mental health) and well-being [3-5], but that it is also useful for mental health promotion, screening, prevention, early intervention, and referral processes [6-9]. Web-based platforms are also being used to deliver real-time stepped care services [10], providing information and low-intensity treatment for those in the early stages of help-seeking, with the capacity to increase scope and intensity of treatments as illnesses progress [11].

In Australia, new and emerging mental health technologies are urgently needed as 1 in 5 young Australians aged between 16 and 24 years experience a mental health disorder each year, yet only 1 in 4 receives professional help [12]. Of those who do receive help, only a small proportion receives timely and evidence-based treatments [13]. Access to quality care is especially difficult for disadvantaged and vulnerable populations, including children; Aboriginal and Torres Strait Islander young peoples; young people from culturally and linguistically diverse backgrounds; and young people living in regional, rural, and remote areas [14,15].

Today, with ever-increasing availability and use of technology, the potential to reach young people is especially promising [7]. For example, in 2014-15, 85% of people aged 15 years and over in Australia were Internet users, and young people in the age group of 15 to 17 years had the highest proportion of use (99%) [16]. Importantly, there is also a growing body of literature supporting the use of Internet-based treatment approaches for a range of mental health problems in adults and adolescents [6,17-19]. Although there is far less research focusing on systems that integrate real-time Web-based stepped care support in mental health services [20], some research examining comprehensive Web-based support systems is starting to emerge [21,22].

Web-Based Health Information

Young people are increasingly relying on the Internet to find answers about their health concerns. For example, in Australia, a national survey in 2012-13 revealed that 53% of young people aged between 16 and 25 years with moderate and very high levels of psychological distress sought Web-based information related to a mental health and/or alcohol or other substance use problem; the majority found this information to be helpful [23]. In other surveys, young people reported feeling comfortable accessing Web-based mental health tools because they felt they were anonymous, welcoming, less stigmatizing, and, for the most part, trustworthy [24]. Similarly, in the United States, a survey in 2013 found that 59% of patients had gone online to look for health information in the previous year and 35% had gone online to self-diagnose or diagnose someone else’s condition. Of the “online diagnosers,” 46% concluded that they needed to see a health professional, whereas 38% preferred a self-management option [25].

Similarly, a small pilot study evaluating an online triage platform also found that, for every user requiring a general practitioner response through an e-consultation, 5 users required online self-help only (WebGP, 2014) [26,27]. The Internet, therefore, offers people seeking information with a useful and easy gateway to answers and solutions that respond to their needs.

Internet-Based Screening

Individuals often show a preference for computerized screening over face-to-face interviews when the subject matter is sensitive in nature [28-32]. Importantly, Internet-based screening for common mental health problems has been shown to be reliable and effective [9,33,34]. In relation to screening for suicidal ideation, it is argued that the implementation of standardized, self-reported, computer-based assessments (with stringent suicide response protocols) may be a strategy that is both accurate and viable if followed up with health professional support [20,35]. In general, there is, however, the potential that symptom, Web-based assessment tools may increase demand on services as the tools are often risk averse (due to medico-legal concerns), recommending professional care when self-management is an appropriate alternative [36,37].


The provision of mental health services though videoconference systems have been widely used since the 1960s [38]. Videoconferencing is viewed as more advantageous than telephone support as the health professional can gauge important visual cues to inform their assessment such as appearance, facial expression, motor activity, movements, and mannerisms [39]. Supportive Web-based conversations with a health professional and referral to appropriate resources have also been found to negate risky behaviors, such as suicide and violence, in highly distressed people [40]. Importalty, videoconferencing has been found to be as reliable as face-to-face assessments, and more cost-effective [41].

In Australia, telepsychiatry (videoconferencing) has been practiced since the 1990s and its benefits have been translated into a wide range of populations including rural populations, Indigenous communities [42], the defense force [43], and children and adolescents [44]. Surprisingly, despite the reliability and benefits of videoconferencing, use of videoconference systems remains low in the different settings [45]. Embedding videoconferencing in comprehensive Web-based support systems would enable a more systematic use of this technology and a greater number of people to access mental health services.

Comprehensive Web-Based Mental Health Care

The augmentation of traditional videoconferencing with online self-reported health questionnaires or screening results has now emerged in the research literature [20,46]. In a study conducted by Williams and colleagues, participants who screened positive for major depression or suicidality were given the opportunity to schedule a videoconference via Skype with a psychiatrist within a 2-week period [46]. In a recent study conducted by our team, an initial Web-based clinical assessment (with an embedded suicidality escalation protocol) was completed as a routine part of the assessment process (either before a video or face-to-face visit with a health professional) for a subset of participants. After the assessment was completed, the severity of mental illness was determined using a clinical staging model [47], and those who were considered to be high-risk (according to the suicidality escalation protocol) were escalated to the youth health service, bringing forward their initial face-to-face appointment [20].

Mental Health Interventions for Young People

In a systematic review of Web-based mental health interventions for young people, however, limited uptake, engagement, adherence, and dropout rates have been identified as significant problems [48,49]. Some researchers, including ourselves, believe that the involvement of young people and youth health professionals in the design, development, and delivery of youth services could lead to better engagement and outcomes [50-52].

Participatory Design and Technology-Based Mental Health Interventions

Participatory design (PD) methodologies, which were developed in the late 1960s and early 1970s, emphasize the importance of involving all stakeholders (including end users, developers, and researchers) during the development of products to help ensure the end product meets everyone’s needs; improve usability; and increase engagement of users [53-55]. The process involves iterative design cycles in which end users and researchers contribute to knowledge production and the development of the end product [50,56]. Importantly, end users should participate in all stages of development [57], not as consultants or controllers of the process, but sharing equal responsibility with the research team for the outcomes [50]. Some researchers [58], including us, consider PD as key research methodologies that overtly put the end user at the center of research and here, the development of the MHeC [59].

The use of PD is expanding in the development of technology-based mental health and well-being interventions for young people. In 2015, a systematic review [50] described the development of these interventions in areas such as prevention, screening [60], and treatment programs [61]. In the majority of cases, however, end users assumed more of a consultative role. Despite the fact that uptake of PD in the development of technology-based mental health and well-being interventions has increased, evidence is still needed to assess the impact of these research methodologies in the outcomes of these interventions [50].

As consumers have the opportunity to share their preferences before the development of expensive, and potentially helpful, systems, the rationale behind the use of PD in the development of mental health technologies could mean that an active engagement of end users could reduce the 17-year gap in translational research [62].

This Study

In Australia, Web-based mental health services include health promotion, self-directed, and low intensity mental health support (eg, ReachOut! [63]; beyondblue [64,65]); national online counseling services (eg, eheadspace [66]); structured self-directed online therapy (eg, MoodGYM [67]); and those offering a combination of assessment and structured online therapy, including additional therapist support (eg, MindSpot Clinic [68]).

What is missing is a Web-based mental health clinic that includes access to all the components and services necessary to meet the needs of all young people (those between the ages of 16 and 25 years), regardless of where they are on the spectrum of mental health or ill-health, or where they reside in Australia.

The aim of this study was to use PD with end users (young people and youth health professionals) and our research team to codesign and build a Mental Health eClinic (MHeC) to improve timely access to, and better quality, mental health care for young people across Australia.


Participants included young people attending headspace Camperdown and headspace Campbelltown, and youth health professionals from both headspace Camperdown and headspace Campbelltown (headspace is the national youth mental health foundation dedicated to improving the well-being of young Australians; Camperdown and Campbelltown are two different sociodemographic areas of Sydney).

The University of Sydney’s Human Research Ethics Committee approved the study (Protocol No. 2014/689). For all phases, participants (young people aged between 16 and 25 years and youth health professionals) who expressed an interest to participate in the study were provided with the participant information statement and participant consent form before providing consent and participating in the study. Parental consent was also obtained for participants aged between 16 and 17 years. At the conclusion of each PD workshop and one-on-one end user consultation, each young person was provided with a gift voucher to thank them for their participation and sharing their expertise; young people were also reimbursed for any travel-related expenses to attend the workshop or session. Youth health professionals were not provided with gift vouchers as they participated in stages 1 and 3 during work time. All workshops were catered.

Recruitment Strategy

Recruitment strategies for young people included the posting of flyers at each headspace center inviting young people to be involved in the study, and informing youth health professionals and reception staff at each headspace center about the study so they too could assist with recruitment. All young people belonging to headspace Camperdown and headspace Campbelltown Youth Reference Group were also invited to participate in the study. Inclusion criteria for the study were as follows: young person attending either headspace Camperdown or headspace Campbelltown; aged between 16 and 25 years; and with access to the Internet through a mobile phone, tablet, desktop, or laptop.

In relation to the recruitment of youth health professionals, senior management at headspace Camperdown and headspace Campbelltown informed all staff about the study and called for expressions of interest to participate.

Research and Development Cycle

The research and development (R&D) cycle for the codesign and build of the MHeC included several iterative PD phases (Figure 1): PD workshops (phase 1); translation of knowledge and ideas generated during workshops (“knowledge translation”; KT) to produce mockups of webpages (either hand-drawn sketches or wireframes; phase 2); and rapid prototyping and one-on-one consultations with end users, including assessing the usability of the online alpha build of the MHeC (phase 3). The remaining phases of PD for the MHeC (rapid prototyping and user [acceptance] testing of the beta build [phase 4], and real-world study of the delta build [phase 5]) are currently underway and will be reported separately.

Participatory Design Workshops (Phase 1)

The PD employed in this study was informed by the research methodologies developed by the Young and Well Cooperative Research Centre for the development of evidence-based online youth mental health promotion, intervention, and treatment [58]. PD workshops were developed to accommodate a maximum of 12 participants per workshop. The same group of researchers facilitated all PD workshops. Importantly, a mental health professional was present on-site for the duration of all workshops involving young people in case anyone experienced psychological distress as a result of the subject matter. A scribe was present to take hand written notes at each workshop.

PD workshops were held in two stages with young people and youth health professionals attending separate workshops (stage 1) or a combined workshop (stage 2; Textbox 1). Following each workshop, the knowledge and ideas generated during the workshop were translated to produce mockups of webpages, either as hand-drawn sketches or wireframes produced using Balsamiq (Balsamiq Solutions, LLC, Sacramento, California, United States: a rapid wireframing tool that reproduces the experience of sketching on a whiteboard but using a computer) [69]. The mockups were then presented at the next workshop, enabling content and broad design ideas to be discussed, critically analyzed, and further developed.

Figure 1. PD research methodologies used during the design and build of the MHeC where end users participate in all stages of development, are at the center of the R&D cycle, and share equal responsibility with the research team for the outcomes. PD: participatory design; MHeC: Mental Health eClinic; R&D: research and development.
View this figure
Phases 1 and 2 workflow.
  • Phase 1, stage 1 (young people only, Camperdown): One 1-day participatory design (PD) workshop with young people attending headspace Camperdown; workshop held in headspace Camperdown

Knowledge translation for young people only (phase 2)

  • Phase 1, stage 1 (young people only, Campbelltown): One 1-day PD workshop with young people attending headspace Campbelltown; workshop held in headspace Campbelltown.

Knowledge translation for youth health professionals only (phase 2)

  • Phase 1, stage 1 (youth health professionals only): One 1-day PD workshop with youth health professionals from headspace Camperdown and headspace Campbelltown; workshop held in headspace Campbelltown.

Knowledge translation combined workshop (phase 2)

  • Phase 1, stage 2 (young people and youth health professionals combined): One half-day PD workshop with both young people and youth health professionals from headspace Camperdown and headspace Campbelltown (all participants must have previously participated in a stage 1 workshop); workshop held in headspace Camperdown.
Textbox 1. Phases 1 and 2 workflow.

Topics for stages 1 and 2 workshops included the MHeC home page; important website functions; look and feel of the website; online physical and mental health assessment; provision of assessment results online (including consideration of a dashboard of results); a video visit system (utilizing live interactive videoconferencing); and the development of personalized well-being plans based on assessment results. Importantly, technology was not used in any PD workshop; instead, several artifacts and design activities (Figure 2) were used in each workshop to facilitate discussions and the design process. The activities included:

  1. The use of propositions to explore and communicate the concept of the MHeC
  2. End user sketching [70] (hand-drawn sketches by young people and youth health professionals, either individually or in groups) to enhance the feedback process and generation of new ideas
  3. Analysis of mockups of webpages (hand-drawn sketches and wireframes) to test designs and provide feedback about the look, feel, content, and behavior of the MHeC.

Knowledge Translation (Phase 2)

At the conclusion of each workshop, the transcript and all visual artifacts were independently analyzed by a KT Team (three 2nd-and 3rd-year psychology students [AI, ML, and ED], all females aged between 20 and 23 years) who were interns at the time at The University of Sydney’s Brain and Mind Centre). Observations were tallied, and those observations with three or more tallies were considered for inclusion in the next generation of wireframes for discussion and analysis at the following workshop. Information was compiled until saturation point was reached (defined as the point where no new information was attained) [71].

One-on-One Consultations With End Users, Including Assessing the Usability of the Alpha Build of the Mental Health eClinic (Phase 3)

Phase 3 involved in-depth one-on-one consultations with new end users (young people and youth health professionals who had not participated in Phase 1). The inclusion of new participants aimed to reduce biased responses due to habituation or familiarity with the topic as a result of prior participation in the study. In each 90-min one-on-one end user consultation, a researcher was paired with a participant (end user) and an observer took notes. Sessions involved the use of laptops where participants had access to the alpha build of the MHeC website. Employing a think-aloud protocol [72], participants were observed as they navigated the MHeC and responded to questions posed by the researcher about the main components of the MHeC; responses were recorded by the observer. The initial effectiveness of the system was then assessed by asking participants to complete 3 usability tasks: (1) create an account and login; (2) find the “need help now” button; and (3) book an appointment. Task completion time was recorded to assess the efficiency of the system. No instructions or clues were provided, and comments in relation to navigation were recorded.

Data Analysis

Qualitative data were interpreted using thematic analysis techniques [73] according to the following themes: general elements of the MHeC; general look and feel; privacy and data sharing; and interaction of the MHeC with social networks. Records of all tallies obtained in Phases 1-3 were then grouped and interpreted by a team of researchers (LOP, TD, and 2nd- and 3rd-year psychology students [AH and FY] who were interns at the time at The University of Sydney’s Brain and Mind Centre). Each theme and associated content was discussed by the group, and differences of opinion were discussed until consensus was reached.

Figure 2. Samples of visual artefacts. (A) Hand-drawn sketch by end users during a PD workshop; (B) Hand-drawn sketch of a webpage generated following a KT session; and (C) Wireframe generated using Balsamiq following a KT session. PD: participatory design; KT: knowledge translation.
View this figure

A total of 4 PD workshops, 4 knowledge translation sessions, and 1 round of one-on-one end user consultation sessions were conducted between October 2014 and June 2015 (Figure 3).

Participant Characteristics

PD workshops were held with young people attending headspace Camperdown (n=7) and headspace Campbelltown (n=11), and youth health professionals working at those services (headspace Camperdown n=5; headspace Campbelltown n=5). A total of 18 young participants participated in stage 1 (young people only) PD workshops: there was equal gender participation, and 78% (14/18) were aged between 18 and 25 years. Ten youth health professionals participated in stage 1 (youth health professionals only) PD workshop: the majority of these participants were female (70%, 7/10) and aged between 20 and 30 years (70%, 7/10). The group comprised 4 psychologists, 2 occupational therapists, 1 medical student, 1 general practitioner, 1 social worker, and 1 Aboriginal youth worker.

Nine participants participated in stage 2 (young people and youth health professionals combined): young people (n=5) and youth health professionals (n=4). The majority of the participants were male (56%, 5/9); the youth health professional’s group included 3 psychologists and 1 occupational therapist.

Six people participated in Phase 3 (consultation with end users, including usability assessment of the online alpha build). The majority of these participants were female (67%, 4/6). The group contained 1 clinical psychologist, 1 psychology student, and 4 young people.

Main Components of the Alpha Build of the Mental Health eClinic

The iterative R&D cycle revealed the importance of five main components of the MHeC. These informed the alpha build of the MHeC (Figure 4-8): a welcoming home page with a visible triage system; a comprehensive physical and mental health assessment; a detailed dashboard of results; a booking and a video visit system; and the generation of a personalized well-being plan that includes links to evidence-based, and health professional-recommended, apps and etools. The five components will have different functionalities depending on whether the user is a young person or a health professional.

Element 1: Home Page and Triage

Young people suggested the home page should be a “welcoming space,” where young people can feel “comfortable,” without compromising the authenticity and professionalism of the site. As such they indicated a banner with the institutional logos should be visible at all times at the footer of the home page. Young people also wanted the following features to be included on the home page: testimonials from young people about their experiences with the MHeC (“a young person explaining why they are there and how to use it”); reliable mental health information (resources); information about how to help a friend; frequently asked questions; and a brief explanation about how the MHeC works. Young people also suggested that as this information would be of interest to a wide range of people, the home page needs to be available to everyone, even if they have not signed up for the MHeC. Youth health professionals and young people suggested that, before any login process, a triage system needs to be in place to ensure that young people in distress can access immediate care and a “need help now” button should be clearly visible for those in crisis.

Figure 3. Study Gantt chart. KT: knowledge translation; MHeC: Mental Health eClinic.
View this figure
Figure 4. Home page with a clearly visible triage system for those requiring urgent help.
View this figure
Figure 5. Comprehensive online physical and mental health assessment.
View this figure
Figure 6. Dashboard of results and progress report.
View this figure
Figure 7. Booking and video visit system.
View this figure
Figure 8. Personalized well-being plan, including recommended apps and etools.
View this figure
Element 2: Online Physical and Mental Health Assessment

Young people said they felt comfortable about completing an online physical and mental health assessment and receiving immediate feedback of their results. The majority also initially reported that they preferred short questionnaires (approximately 15 min in duration). After an explanation was provided, however, about the number and types of questionnaires that would be included to fully assess their physical and mental health (including the range of standardized assessments that are currently being used in headspace centers and completed using paper and pencil or iPad) and that a comprehensive assessment would enable a detailed dashboard of results and well-being plan to be generated (including recommended apps and etools), young people understood the need to complete a more comprehensive assessment (up to 1-hour duration). Young people suggested that a pause and “resume later” feature would be helpful with a longer questionnaire to ensure completion. In addition, young people reported that they preferred Likert-scale questions (typically a 5-, 7-, or 9-point agreement scale), two-way close-ended question (no/yes) or multiple-choice questions rather than free-response (“enter text”) questions. The MHeC’s assessment includes medical history, physical health, mental health symptomatology, and health behaviors. For youth health professionals, the questionnaire functionality enables them to write notes or complete/add additional information for relevant questions in relation to their young person’s progress or specific clinical observations (eg, provide a score in the Social and Occupational Functioning Assessment Scale [74] or allocate participants to a clinical stage [47]).

Element 3: Dashboard of Results and Progress

Young people reported that after completing the assessment, immediate feedback about their results was essential. They also reported that knowing this would occur would improve motivation to respond as best as possible and answer all questions. Young people said they wanted accurate feedback about their results and for this to be represented in a detailed dashboard (“visual concept”), with the option of printing or saving the file as a PDF for their records, or to share with a health professional in the future. Simple bar graphs, colored icons, and traffic light representations were seen as the most acceptable and understandable options for the presentation of results. Line graphs were the preferred option to represent changes over time and to track progress. Young people also suggested the availability of a “customize option” would be useful to enable individuals to select multiple variables of interest to explore if, and how, they influence another. Youth health professionals would have access to a young person’s results and progress as they reported that a dashboard of results would help guide care before and during video and face-to-face visits.

Element 4: Waiting Room, Booking System, and Video Visit System

Young people requested that a booking system be available to make timely appointments with youth health professionals attached to the clinic. Young people also suggested a “waiting room” be included in the MHeC where individuals wait for their video visit to begin. While on this page, young people suggested various activities could be available, for example, breathing exercises or watching selected videos until the video visit icon changes color, signaling their video visit is about to commence. In relation to the youth health professionals’ functionality of this component, they suggested that it would be important to have access to a booking system (“to add or cancel a booking as well as block timeslots”).

Acknowledging cultural preferences, and that some people do not feel comfortable seeing their image (“seeing myself distracts me, I would be looking at myself”), it was suggested by some young people that at the moment of booking an appointment the option of hiding their image should be available. It was also suggested that in the event of communication cut-offs or a young person wanting to say something sensitive, a chat box would allow a fluid conversation. All participants agreed that the video visit system needs to be embedded in the MHeC as “it would be secure.” As it happens in regular practice, the youth health professionals are in charge of inviting a young person to come to their office; therefore, in the MHeC, the youth health professional has the possibility to start and end the video visit. Importantly, this video visit system includes a “share” functionality, where relevant information can be shared and made visible on a young person’s screen (eg, “explain their dashboard of results during the session”).

Element 5: Personalized Well-Being Plan That Includes Links to Evidence-Based, and Health Professional–Recommended, Apps and Etools

All participants reported that they would like to be provided with a personalized well-being plan (generated from their assessment results) that included tailored automatic recommendations on how to improve their health and well-being. Young people believe that apps and etools targeting their main concerns/issues would be beneficial. However, young people believe that “health professional recommended” and free apps/etools are more likely to be downloaded over apps/etools that have to be purchased or that are not recommended by health professionals. The areas of main interest to young people were apps/etools that help them with their sleep; improve their memory; assist with mood tracking; and help with tracking their progress over time. Ideally, apps and wearable devices should be integrated with the MHeC. Youth health professionals felt confident about the MHeC suggesting apps and etools to young people “only if they are from a reputable source” and had greater confidence in those that had been independently rated using the Mobile App Rating Scale [75]. In terms of functionality, all end users suggested that the MHeC should have a list of apps sorted by categories such as sleep, mood, anxiety, and physical health, among others. Participants suggested that each app should have an Apple Store and Google Play link to facilitate the download process.

User Interface

In relation to the look and feel of the MHeC, young people and youth health professionals agree that the system should be “clean and tidy,” displaying short and concise information and making good use of space. Most of the participants preferred to have icons instead of text and preferred to self-explore the system rather than having lengthy instructions. The use of stereotyped photos with “happy” or “sad people” was highly discouraged by young people. The participants wanted to see videos about how to use the system, but indicated that these should be short (no more than 30 s) and not auto-play due to data download size concerns and privacy issues (eg, a video commencing while sitting in a public space). Soothing colors were preferred for the background and brighter colors for the functionalities, suggesting combinations such as blue and orange for consideration. Young people also suggested the possibility of a “customizing” option to enable users to choose between two to three text types and background color options.

Privacy and Data Sharing

Young males were more concerned about privacy issues than young females. For example, young male participants said “sharing data is OK, but it must say it's confidential” or “I wouldn’t share my location with the system.” On the other hand, young female participants said, for example, “I have no privacy concerns specifically if going to a clinician.” All young people, however, said they would share their data as they believe the MHeC would be a professional and trustworthy site and this would enable them “to get most help.” Young people, however, emphasized that they want to be informed about how the data will be handled before giving permission for data sharing and to have the opportunity to withdraw this permission at any time.

Integration With Social Networks

Young people believe that the MHeC should be part of social networks as they would like to be able to share the MHeC in their profile or with a friend through a private message. Moreover, they wanted to be able to like the MHeC or write a comment on it. One of the suggested preferred features of the MHeC was “tips” and for these to be displayed throughout all components of the MHeC. Young people also thought it would be valuable if they were able to share these tips via their social networks.


Information about the efficiency and effectiveness of the alpha build of the MHeC was obtained during the one-on-one consultation sessions. All participants (n=6) completed usability tasks. All participants found that “creating a MHeC account” and “login” were relatively simple processes. Half of the participants thought that giving the option of login with their social media details (single sign on) was a good idea as it would speed up this process (“super easy”). The other half were against social media login as they were concerned about the MHeC sharing information with their friends (“I don’t want my friends seeing this information”). Finding the “need help now” button was a simple task; all participants found it in less than 5 s. Booking an appointment was a straightforward task as well, and participants were able to navigate the MHeC and complete the process almost immediately.

Principal Findings

Our study utilized an innovative approach to the development of a Web-based mental health clinic for young people (the MHeC). The PD employed ensured that end users (young people and youth health professionals) had an active and equal involvement in all phases of the design and development process. By engaging these stakeholders, we attempted to respond to end users’ expectations about the MHeC and what was achievable in terms of technology and what has been proven to be effective in the mental health field. The use of several design activities (propositions, hand-drawn sketches, and mockups) and the combination of different PD methodologies (workshops and one-on-one consultations with end users, including assessing usability of the online alpha build of the MHeC) enhanced feedback processes and the generation of new ideas. This combination of research methodologies also accelerated the refinement of the MHeC to achieve the build of the alpha prototype.

In the past decades, the development and use of eHealth solutions in mental health care have expanded; however, these solutions have been developed to address specific problems or to replace different components of the traditional health care system. As an example, the majority of self-triage tools rely on people actively searching for these tools on the Internet; however, some health services provide self-triage tools on their websites, particularly when booking appointments online [26]. Telepsychiatry (videoconferencing), as another example, has also been a particularly effective way of providing support as it allows real-time interaction while negating barriers such as cost, geographical location, and stigma concerns associated with face-to-face support [46]. To the best of our knowledge, our MHeC is one of a kind as it integrates triage, online assessments, online provision of results with easy-to-interpret graphic representations (a dashboard), enables video visits with youth health professionals, provides a personalized well-being plan with immediate interventions, and tracks progress. We believe this is more sophisticated and technologically advanced than traditional telemedicine.

One of the strengths of our system is that it integrates new and emerging technologies with the traditional face-to-face process. Several studies emphasize that screening alone is insufficient for connecting end users with the necessary resources for effective treatment [9,76-78]. Instead, it is argued that online screening should supplement, and be integrated when necessary, with additional support and assessment, which can include face-to-face or online assessments with health professionals within mental health services [9,78]. In line with these recommendations, Internet-based clinical assessments have been implemented and trialed in various Web-based clinics as a means of rapid assessment and referral to appropriate online interventions [79,80] and integrated with face-to-face and online support [20].

Important considerations need to be made in relation to an end user’s health literacy as it is argued that Web-based health care communication demands a higher level of health literacy from end users, including the knowledge and skills that enable them to navigate the health system [81]. National datasets report that 60% of Australians have health literacy scores at the lowest two levels of proficiency (scoring 1 or 2), whereas only 6% attain the highest two levels (scoring 4 or 5) [82]. Furthermore, if an end user is experiencing active psychosis or is in crisis, for example, the ability to accurately read or reflect upon information may prove challenging [83,84].

Video visits (as provided in the MHeC with “share” functionality), and face-to-face support provided by health professionals, may provide end users with the support they need to help them navigate the online physical and mental health assessment, dashboard of results, and the health system. These noted challenges highlight the need for health professionals to remain involved in some capacity in the online assessment process, particularly to prevent the unnecessary funneling of end users into the health system.

In 2017, it was estimated 95% of young people in Australia had access to a mobile phone [85,86], making Internet access more widely available to individuals. Mobile phones also provide individuals with higher levels of privacy (compared with shared computers) when participating in a video visit or completing assessments. Consequently, these devices could serve as great facilitators of health care provision, playing a crucial role in health reform.


In 2014, the Australian Government asked the National Mental Health Commission (NMHC) to review existing mental health services and programs and “…assess the efficiency and effectiveness of programs and services in supporting individuals experiencing mental ill health and their families and other support people to lead a contributing life and to engage productively in the community” [87]. Similar to previous reviews, the NMHC, called for an overhaul of the Australian mental health system, including an integration of e-mental health with face-to-face services. Innovations in the use of technology during the assessment process hold potential as they can go some way to addressing documented youth service capacity issues [84,88].

In alignment with the NMHC’s three key components (person-centered design principles; new system architecture; and shifting funding to more efficient and effective “upstream” of services and supports), we envisage that the MHeC will result in a usable system, with high engagement rates, and could finally improve access for all young people requiring assistance across Australia. We see the MHeC as a real-time primary care clinic integrated with current face-to-face services, offering end users at a minimum immediate online clinical assessment; immediate generation of a dashboard of results and personalized well-being plan with tailored interventions; and inbuilt triage and escalation protocols to accurately respond to severe and risky cases. The MHeC offers timely support and interventions through the development of an individual and health professional shared treatment plan. It also provides an opportunity for young people to have a video visit with a health professional despite their current geographical location. Although developed for Australian settings, the MHeC has the potential to be customized for use in developed and developing countries (and especially in those countries where Internet connectivity is high).

Information obtained from this alpha build (more specifically it’s screening and triage protocols; online physical and mental health assessment, video visit system; and self-directed mental health support features) are now informing the development and build of the Synergy Online System (Synergy). Synergy is a Web-based modular platform that links integrated and interoperable resources (eg, apps, etools, data sharing, and access to online and in-clinic health services). Synergy operates through existing health providers to promote access to high-quality and cost-effective mental health services. This system does not deliver services or compete with existing service providers; rather, it aims to complement them by linking with other services. In addition, it enables real-time health and social outcomes tracking, thus providing high-quality and personalized service recommendations to the person seeking help. Synergy has been configured to permit the transfer of individual-level data (allowing for other ethical, consent, governance, and privacy considerations) between it and other existing record systems. The actual capacity to do this in any specific service setting depends on the ways in which existing service systems are configured and governed. Synergy aims to enable Australian mental health service transformation for better outcomes for individual users, their families and supportive others, health professionals, and service providers.

Currently, a series of collaborative research trials are planned to evaluate the use of Synergy across the lifespan. This research is funded by a 3-year agreement (2017-20) between the Australian Government Department of Health and InnoWell Pty Ltd (a joint venture between The University of Sydney and PricewaterhouseCoopers) to the value of Aus $30 M. Results from this research will be reported separately.


Our sample size in Phase 3 (one-on-one consultation with end users) was in the lower range of the average numbers for this type of study (between 6 and 12 participants) [89]. However, our sample size still enabled us to collect sufficient information for analysis in the framework and reach a saturation point. Further research is needed to understand the acceptability and usability of the system, as well as to validate all the components in real-world settings.


The MHeC has been designed to be accessible, affordable, available, and applicable to the many and varied settings in which there is the opportunity to connect with young people experiencing mental health concerns at anytime, anywhere across Australia, regardless of geographic location. The MHeC especially represents a solution for regional and rural/remote locations and the developing world where the need for mental health care far outstrips the number of health professionals available.

To the best of our knowledge, it is the first time that a Web-based clinic has been conceptualized and codesigned through an iterative cycle with end users. The close collaboration between all different stakeholders has the potential to effectively increase accessibility to mental health services for young people, the quality of care provided to young people, and also increase engagement with, and usability of, the final product. The main components of this system ensure that young people seeking help can find what they need within one system. The alpha prototype of the MHeC not only provides end users with accurate health information but also provides each person with the opportunity to have a comprehensive clinical (physical and mental health) assessment as well as real-time tailored interventions, including a video visit with an appropriate heath professional. Further research is needed to assess the acceptability of the MHeC as well as its applicability in large-scale studies.


The authors would like to thank all the participants in the study from headspace Camperdown and headspace Campbelltown for their collaboration; Ms Ashling Isik and Ms Min-Shi Michelle Lim for analyzing and mocking up the data; Ms Ellena Daniele for analyzing and representing the data throughout the entire process; and Ms Anna Maria Hanna and Mr Fouad Yasin for assisting in the data analysis. The authors would also like to thank Ms Antonia Ottavio (Clinical Services Manager, headspace Camperdown) and Dr Shane Cross (Clinical Services Manager, headspace Campbelltown) for supporting this study during 2014-15. This project was funded by the Young and Well Cooperative Research Centre, which was led by Professor Jane Burns.

Conflicts of Interest

LOP, TAD, CSR, and ACM have no conflicts of interest to declare. EMS is the Medical Director of the Young Adult Mental Health Unit, St Vincent’s Hospital, Darlinghurst; Discipline Leader of Adult Mental Health, School of Medicine, University of Notre Dame; Research Affiliate, The University of Sydney; and Consultant Psychiatrist. She has received honoraria for educational seminars related to the clinical management of depressive disorders supported by Servier and Eli-Lilly pharmaceuticals. She has participated in a national advisory board for the antidepressant compound Pristiq, manufactured by Pfizer. She was the National Coordinator of an antidepressant trial sponsored by Servier. IBH has been a Mental Health Commissioner at the National Mental Health Commission since 2012. He is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC), University of Sydney. The BMC operates an early-intervention youth service at Camperdown under contract to headspace. IBH has previously led community-based and pharmaceutical industry–supported (Wyeth, Eli Lily, Servier, Pfizer, AstraZeneca) projects focused on the identification and better management of anxiety and depression. He is a member of the Medical Advisory Panel for Medibank Private, a Board Member of Psychosis Australia Trust, and a member of the Department of Veterans’ Affairs Veterans Mental Health Clinical Reference Group. He is the Chief Scientific Advisor to, and an equity shareholder in, Innowell Pty Ltd. Innowell has been formed by The University of Sydney and PricewaterhouseCoopers (PwC) to deliver the Aus $30 M Australian Government-funded “Project Synergy.” Project Synergy is a 3-year program for the transformation of mental health services through the use of innovative technologies.

  1. McHugh RK, Barlow DH. The dissemination and implementation of evidence-based psychological treatments. A review of current efforts. Am Psychol 2010;65(2):73-84. [CrossRef] [Medline]
  2. Judd FK, Jackson H, Davis J, Cockram A, Komiti A, Allen N, et al. Improving access for rural Australians to treatment for anxiety and depression: The University of Melbourne Depression and Anxiety Research and Treatment Group-Bendigo Health Care Group initiative. Aust J Rural Health 2001 Apr;9(2):91-96. [Medline]
  3. Eysenbach G, Powell J, Kuss O, Sa E. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. J Am Med Assoc 2002;287(20):2691-2700. [Medline]
  4. Kauer SD, Mangan C, Sanci L. Do online mental health services improve help-seeking for young people? A systematic review. J Med Internet Res 2014;16(3):e66 [FREE Full text] [CrossRef] [Medline]
  5. Wood FB, Benson D, LaCroix EM, Siegel ER, Fariss S. Use of Internet audience measurement data to gauge market share for online health information services. J Med Internet Res 2005 Jul 01;7(3):e31 [FREE Full text] [CrossRef] [Medline]
  6. Griffiths KM, Farrer L, Christensen H. The efficacy of internet interventions for depression and anxiety disorders: a review of randomised controlled trials. Med J Aust 2010 Jun 7;192(11 Suppl):S4-11. [Medline]
  7. Burns JM, Davenport TA, Christensen H, Luscombe GM, Mendoza JA, Bresnan A, et al. Game On: Exploring the Impact of Technologies on Young Men's Mental Health and Wellbeing: Findings from the first Young and Well National Survey. Melbourne: Young and Well Cooperative Research Centre; 2013.
  8. Oh E, Jorm AF, Wright A. Perceived helpfulness of websites for mental health information: a national survey of young Australians. Soc Psychiatry Psychiatr Epidemiol 2009 Apr;44(4):293-299. [CrossRef] [Medline]
  9. Kim EH, Coumar A, Lober WB, Kim Y. Addressing mental health epidemic among university students via web-based, self-screening, and referral system: a preliminary study. IEEE Trans Inf Technol Biomed 2011 Mar;15(2):301-307. [CrossRef] [Medline]
  10. Cross SP, Hickie I. Transdiagnostic stepped care in mental health. Public Health Res Pract 2017 Apr 27;27(2):- [FREE Full text] [CrossRef] [Medline]
  11. Kauer SD, Mangan C, Sanci L. Do online mental health services improve help-seeking for young people? A systematic review. J Med Internet Res 2014;16(3):e66 [FREE Full text] [CrossRef] [Medline]
  12. Slade T, Johnston A, Oakley Browne MA, Andrews G, Whiteford H. 2007 National Survey of Mental Health and Wellbeing: methods and key findings. Aust N Z J Psychiatry 2009 Jul;43(7):594-605. [CrossRef] [Medline]
  13. Libby AM, Brent DA, Morrato EH, Orton HD, Allen R, Valuck RJ. Decline in treatment of pediatric depression after FDA advisory on risk of suicidality with SSRIs. Am J Psychiatry 2007 Jun;164(6):884-891. [CrossRef] [Medline]
  14. Clarke G, Yarborough BJ. Evaluating the promise of health IT to enhance/expand the reach of mental health services. Gen Hosp Psychiatry 2013;35(4):339-344 [FREE Full text] [CrossRef] [Medline]
  15. Dudgeon P, Walker R, Scrine C, Shepherd C, Calma T, Ring I. Effective Strategies to Strengthen the Mental Health and Wellbeing of Aboriginal and Torres Strait Islander People. Canberra: Australian Institute of Health and Welfare (AIHW); 2014:1-72.
  16. Australian Bureau of Statistics. 2014. 8146.0 Household Use of Information Technology, Australia, 2016-17   URL: [WebCite Cache]
  17. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther 2009;38(4):196-205. [CrossRef] [Medline]
  18. Naslund JA, Marsch LA, McHugo GJ, Bartels SJ. Emerging mHealth and eHealth interventions for serious mental illness: a review of the literature. J Ment Health 2015;24(5):321-332. [CrossRef] [Medline]
  19. Tate DF, Finkelstein EA, Khavjou O, Gustafson A. Cost effectiveness of internet interventions: review and recommendations. Ann Behav Med 2009 Aug;38(1):40-45 [FREE Full text] [CrossRef] [Medline]
  20. Iorfino F, Davenport TA, Ospina-Pinillos L, Hermens DF, Cross S, Burns J, et al. Using new and emerging technologies to identify and respond to suicidality among help-seeking young people: a cross-sectional study. J Med Internet Res 2017 Jul 12;19(7):e247 [FREE Full text] [CrossRef] [Medline]
  21. Frost J, Okun S, Vaughan T, Heywood J, Wicks P. Patient-reported outcomes as a source of evidence in off-label prescribing: analysis of data from PatientsLikeMe. J Med Internet Res 2011;13(1):e6 [FREE Full text] [CrossRef] [Medline]
  22. Faurholt-Jepsen M, Vinberg M, Frost M, Christensen EM, Bardram J, Kessing LV. Daily electronic monitoring of subjective and objective measures of illness activity in bipolar disorder using smartphones--the MONARCA II trial protocol: a randomized controlled single-blind parallel-group trial. BMC Psychiatry 2014;14:309 [FREE Full text] [CrossRef] [Medline]
  23. Burns JM, Birrell E, Bismark M, Pirkis J, Davenport TA, Hickie IB, et al. The role of technology in Australian youth mental health reform. Aust Health Rev 2016 Dec;40(5):584-590. [CrossRef] [Medline]
  24. Powell J, Clarke A. Investigating internet use by mental health service users: interview study. Stud Health Technol Inform 2007;129(Pt 2):1112-1116. [Medline]
  25. Fox S, Duggan M. Pew Research Center. 2013. Health Online 2013   URL: [WebCite Cache]
  26. Castle-Clarke S, Imison C. The digital patient: transforming primary care?. London: Nuffield Trust; 2016 Nov.   URL: [WebCite Cache]
  27. Madan A. GP Access. 2014. webGP: the virtual general practice   URL: [WebCite Cache]
  28. Greist JH, Gustafson DH, Stauss FF, Rowse GL, Laughren TP, Chiles JA. A computer interview for suicide-risk prediction. Am J Psychiatry 1973 Dec;130(12):1327-1332. [CrossRef] [Medline]
  29. Hines DA, Douglas EM, Mahmood S. The effects of survey administration on disclosure rates to sensitive items among men: a comparison of an Internet panel sample with a RDD telephone sample. Comput Human Behav 2010 Nov 01;26(6):1327-1335 [FREE Full text] [CrossRef] [Medline]
  30. Heeren T, Edwards EM, Dennis JM, Rodkin S, Hingson RW, Rosenbloom DL. A comparison of results from an alcohol survey of a prerecruited Internet panel and the National Epidemiologic Survey on Alcohol and Related Conditions. Alcohol Clin Exp Res 2008 Feb;32(2):222-229. [CrossRef] [Medline]
  31. Tourangeau R, Yan T. Sensitive questions in surveys. Psychol Bull 2007 Sep;133(5):859-883. [CrossRef] [Medline]
  32. Bradford S, Rickwood D. Acceptability and utility of an electronic psychosocial assessment (myAssessment) to increase self-disclosure in youth mental healthcare: a quasi-experimental study. BMC Psychiatry 2015 Dec 01;15:305 [FREE Full text] [CrossRef] [Medline]
  33. Cuijpers P, Boluijt P, van Straten A. Screening of depression in adolescents through the Internet : sensitivity and specificity of two screening questionnaires. Eur Child Adolesc Psychiatry 2008 Feb;17(1):32-38. [CrossRef] [Medline]
  34. Farvolden P, McBride C, Bagby RM, Ravitz P. A Web-based screening instrument for depression and anxiety disorders in primary care. J Med Internet Res 2003;5(3):e23 [FREE Full text] [CrossRef] [Medline]
  35. Lawrence ST, Willig JH, Crane HM, Ye J, Aban I, Lober W, et al. Routine, self-administered, touch-screen, computer-based suicidal ideation assessment linked to automated response team notification in an HIV primary care setting. Clin Infect Dis 2010 Apr 15;50(8):1165-1173 [FREE Full text] [CrossRef] [Medline]
  36. Lupton D, Jutel A. 'It's like having a physician in your pocket!' A critical analysis of self-diagnosis smartphone apps. Soc Sci Med 2015 May;133:128-135. [CrossRef] [Medline]
  37. Semigran HL, Linder JA, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagnosis and triage: audit study. Br Med J 2015;351:h3480 [FREE Full text] [Medline]
  38. Shore J. The evolution and history of telepsychiatry and its impact on psychiatric care: current implications for psychiatrists and psychiatric organizations. Int Rev Psychiatry 2015;27(6):469-475. [CrossRef] [Medline]
  39. Cukor P, Baer L, Willis BS, Leahy L, O'Laughlen J, Murphy M, et al. Use of videophones and low-cost standard telephone lines to provide a social presence in telepsychiatry. Telemed J 1998;4(4):313-321. [CrossRef] [Medline]
  40. Barak A. Emotional support and suicide prevention through the Internet: a field project report. Comput Hum Behav 2007 Mar;23(2):971-984. [CrossRef]
  41. Hubley S, Lynch SB, Schneck C, Thomas M, Shore J. Review of key telepsychiatry outcomes. World J Psychiatry 2016 Jun 22;6(2):269-282 [FREE Full text] [CrossRef] [Medline]
  42. Alexander J, Lattanzio A. Utility of telepsychiatry for Aboriginal Australians. Aust N Z J Psychiatry 2009 Dec;43(12):1185. [CrossRef] [Medline]
  43. Wallace D, Rayner S. Telepsychiatry services in the Australian Defence Force. Australas Psychiatry 2013 Jun;21(3):278-279. [CrossRef] [Medline]
  44. Wood J, Stathis S, Smith A, Krause J. E-CYMHS: an expansion of a child and youth telepsychiatry model in Queensland. Australas Psychiatry 2012 Aug;20(4):333-337. [CrossRef] [Medline]
  45. Douglas MD, Xu J, Heggs A, Wrenn G, Mack DH, Rust G. Assessing telemedicine utilization by using Medicaid claims data. Psychiatr Serv 2017 Feb 01;68(2):173-178 [FREE Full text] [CrossRef] [Medline]
  46. Williams A, Larocca R, Chang T, Trinh N, Fava M, Kvedar J, et al. Web-based depression screening and psychiatric consultation for college students: a feasibility and acceptability study. Int J Telemed Appl 2014;2014:580786 [FREE Full text] [CrossRef] [Medline]
  47. Cross SP, Hermens DF, Scott EM, Ottavio A, McGorry PD, Hickie IB. A clinical staging model for early intervention youth mental health services. Psychiatr Serv 2014 Jul;65(7):939-943. [CrossRef] [Medline]
  48. Clarke AM, Kuosmanen T, Barry MM. A systematic review of online youth mental health promotion and prevention interventions. J Youth Adolesc 2015 Jan;44(1):90-113. [CrossRef] [Medline]
  49. Doherty G, Coyle D, Sharry J. Engagement with Online Mental Health Interventions: An Exploratory Clinical Study of a Treatment for Depression. 2012 Presented at: ACM Annual Conference on Human Factors in Computing Systems - CHI '12; May 5-10, 2012; Austin   URL:
  50. Orlowski SK, Lawn S, Venning A, Winsall M, Jones GM, Wyld K, et al. Participatory research as one piece of the puzzle: a systematic review of consumer involvement in design of technology-based youth mental health and well-Being interventions. JMIR Hum Factors 2015;2(2):e12 [FREE Full text] [CrossRef] [Medline]
  51. Lyles CR, Altschuler A, Chawla N, Kowalski C, McQuillan D, Bayliss E, et al. User-centered design of a tablet waiting room tool for complex patients to prioritize discussion topics for primary care visits. JMIR Mhealth Uhealth 2016 Sep 14;4(3):e108 [FREE Full text] [CrossRef] [Medline]
  52. Bilsbury CD, Richman A. A staging approach to measuring patient-centred subjective outcomes. Acta Psychiatr Scand Suppl 2002(414):5-40. [Medline]
  53. Bjerkan J, Hedlund M, Hellesø R. Patients' contribution to the development of a web-based plan for integrated care - a participatory design study. Inform Health Soc Care 2015 Mar;40(2):167-184. [CrossRef] [Medline]
  54. Rothgangel A, Braun S, Smeets R, Beurskens A. Design and development of a telerehabilitation platform for patients with phantom limb pain: a user-centered approach. JMIR Rehabil Assist Technol 2017 Feb 15;4(1):e2 [FREE Full text] [CrossRef] [Medline]
  55. Peters D, Davis S, Calvo RA, Sawyer SM, Smith L, Foster JM. Young people's preferences for an asthma self-management app highlight psychological needs: a participatory study. J Med Internet Res 2017 Apr 11;19(4):e113 [FREE Full text] [CrossRef] [Medline]
  56. Muller MJ, Kuhn S. Participatory design. Commun ACM 1993;36(6):24-28. [CrossRef]
  57. Sjöberg C, Timpka T. Participatory design of information systems in health care. J Am Med Inform Assoc 1998;5(2):177-183 [FREE Full text] [Medline]
  58. Hagen P, Collin P, Metcalf A, Nicholas M, Rahilly K, Swainston N. Participatory design of evidence-based online youth mental health promotion, intervention and treatment: A Young and Well Cooperative Research Centre Innovative Methodologies Guide. Melbourne: Young and Well Cooperative Research Centre; 2012.   URL: https:/​/www.​​__data/​assets/​pdf_file/​0005/​476330/​Young_and_Well_CRC_IM_PD_Guide.​pdf [WebCite Cache]
  59. Sullivan P. Beyond a narrow conception of usability testing. IEEE Trans Profess Commun 1989;32(4):256-264. [CrossRef]
  60. Gordon M, Henderson R, Holmes JH, Wolters MK, Bennett IM. Participatory design of ehealth solutions for women from vulnerable populations with perinatal depression. J Am Med Inform Assoc 2016 Jan;23(1):105-109. [CrossRef] [Medline]
  61. Monshat K, Vella-Brodrick D, Burns J, Herrman H. Mental health promotion in the Internet age: a consultation with Australian young people to inform the design of an online mindfulness training programme. Health Promot Int 2012 Jun;27(2):177-186 [FREE Full text] [CrossRef] [Medline]
  62. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med 2011 Dec;104(12):510-520 [FREE Full text] [CrossRef] [Medline]
  63. Burns J, Morey C, Lagelée A, Mackenzie A, Nicholas J. Reach Out! Innovation in service delivery. Med J Aust 2007 Oct 1;187(7 Suppl):S31-S34. [Medline]
  64. Harman G, Heath J. Australian country perspective: the work of beyondblue and SANE Australia. In: Gaebel W, Rössler W, Sartorius N, editors. The Stigma of Mental Illness - End of the Story?. Cham, Switzerland: Springer International Publishing; 2017:289-315.
  65. Hickie I. Can we reduce the burden of depression? The Australian experience with beyondblue: the national depression initiative. Australas Psychiatry 2004;12 Suppl:S38-S46. [CrossRef] [Medline]
  66. Rickwood D, Webb M, Kennedy V, Telford N. Who are the young people choosing web-based mental health support? Findings from the implementation of Australia's national web-based youth mental health service, eheadspace. JMIR Ment Health 2016 Aug 25;3(3):e40 [FREE Full text] [CrossRef] [Medline]
  67. Christensen H, Griffiths KM, Korten A. Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scores. J Med Internet Res 2002;4(1):e3 [FREE Full text] [CrossRef] [Medline]
  68. Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, et al. MindSpot Clinic: an accessible, efficient, and effective online treatment service for anxiety and depression. Psychiatr Serv 2015 Oct;66(10):1043-1050. [CrossRef] [Medline]
  69. Balsamiq. 2017. Balsamiq Studios   URL: [accessed 2017-12-04] [WebCite Cache]
  70. Buxton B. Sketching User Experiences: Getting the Design Right and the Right Design. Burlington: Morgan Kaufmann; 2007:1-448.
  71. Guest G, Bunce A, Johnson L. How many interviews are enough?: an experiment with data saturation and variability. Field Methods 2006 Feb 01;18(1):59-82. [CrossRef]
  72. Boren T, Ramey J. Thinking aloud: reconciling theory and practice. IEEE Trans Profess Commun 2000;43(3):261-278. [CrossRef]
  73. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Jan;3(2):77-101. [CrossRef]
  74. Goldman HH, Skodol AE, Lave TR. Revising axis V for DSM-IV: a review of measures of social functioning. Am J Psychiatry 1992 Sep;149(9):1148-1156. [Medline]
  75. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3(1):e27 [FREE Full text] [CrossRef] [Medline]
  76. Eisenberg D, Gollust SE, Golberstein E, Hefner JL. Prevalence and correlates of depression, anxiety, and suicidality among university students. Am J Orthopsychiatry 2007 Oct;77(4):534-542. [CrossRef] [Medline]
  77. Garlow SJ, Rosenberg J, Moore JD, Haas AP, Koestner B, Hendin H, et al. Depression, desperation, and suicidal ideation in college students: results from the American Foundation for Suicide Prevention College Screening Project at Emory University. Depress Anxiety 2008;25(6):482-488. [CrossRef] [Medline]
  78. Gilbody S, Sheldon T, House A. Screening and case-finding instruments for depression: a meta-analysis. CMAJ 2008 Apr 08;178(8):997-1003 [FREE Full text] [CrossRef] [Medline]
  79. Donker T, Griffiths KM, Cuijpers P, Christensen H. Psychoeducation for depression, anxiety and psychological distress: a meta-analysis. BMC Med 2009;7:79 [FREE Full text] [CrossRef] [Medline]
  80. Klein MC, Ciotoli C, Chung H. Primary care screening of depression and treatment engagement in a university health center: a retrospective analysis. J Am Coll Health 2011;59(4):289-295. [CrossRef] [Medline]
  81. Baum F, Lawless A, Delany T, Macdougall C, Williams C, Broderick D, et al. Evaluation of health in all policies: concept, theory and application. Health Promot Int 2014 Jun;29 Suppl 1:i130-i142. [CrossRef] [Medline]
  82. Australian Bureau of Statistics. 2008. Adult Literacy and Life Skills Survey, Summary Results, Australia, 2006 (Reissue)   URL: [WebCite Cache]
  83. Suler J. Assessing a person's suitability for online therapy: the ISMHO clinical case study group. International Society for Mental Health Online. Cyberpsychol Behav 2001 Dec;4(6):675-679. [CrossRef] [Medline]
  84. Rickwood DJ, Telford NR, Parker AG, Tanti CJ, McGorry PD. headspace - Australia's innovation in youth mental health: who are the clients and why are they presenting? Med J Aust 2014 Feb 3;200(2):108-111. [Medline]
  85. Roymorgan. 9 in 10 Aussie teens now have a mobile (and most are already on to their second or subsequent handset)   URL: http:/​/www.​​findings/​6929-australian-teenagers-and-their-mobile-phones-june-2016-201608220922 [accessed 2018-02-19] [WebCite Cache]
  86. Deloitte. Smart everything, everywhere Mobile Consumer Survey 2017 The Australian cut   URL: [accessed 2018-02-19] [WebCite Cache]
  87. National Mental Health Commission. Report of the National Review of Mental Health Programmes and Services: Summary. Sydney: National Mental Health Commission; 2014.
  88. Rickwood D, Van Dyke N, Telford N. Innovation in youth mental health services in Australia: common characteristics across the first headspace centres. Early Interv Psychiatry 2015 Feb;9(1):29-37. [CrossRef] [Medline]
  89. Dumas JS, Redish J. A Practical Guide to Usability Testing. Exeter: Intellect Books; 1999.

KT: knowledge translation
MHeC: Mental Health eClinic
NMHC: National Mental Health Commission
PD: participatory design
R&D: research and development

Edited by J Torous, G Eysenbach; submitted 22.12.17; peer-reviewed by J van Bebber, IN Gomez, C De Freitas-Vidal, R Pankomera, G Strudwick; comments to author 05.02.18; revised version received 17.03.18; accepted 22.03.18; published 28.05.18


©Laura Ospina-Pinillos, Tracey A Davenport, Cristina S Ricci, Alyssa C Milton, Elizabeth M Scott, Ian B Hickie. Originally published in the Journal of Medical Internet Research (, 28.05.2018.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.