This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Globally there is increasing recognition that new strategies are required to reduce disability due to common mental health problems. As 75% of mental health and substance use disorders emerge during the teenage or early adulthood years, these strategies need to be readily accessible to young people. When considering how to provide such services at scale, new and innovative technologies show promise in augmenting traditional clinic-based services.
The aim of this study was to test new and innovative technologies to assess clinical stage in early intervention youth mental health services using a prototypic online system known as the Mental Health eClinic (MHeC).
The online assessment within the MHeC was compared directly against traditional clinician assessment within 2 Sydney-based youth-specific mental health services (
Of the 72 participants who completed the study, 71% (51/72) were female and the mean age was 20.4 years (aged 16 to 25 years); 68% (49/72) of participants were recruited from
The MHeC presents a new and innovative method for determining key clinical service parameters. It has the potential to be adapted to varied settings in which young people are connected with traditional clinical services and assist in providing the right care at the right time.
Globally, there is increasing recognition that new strategies are required to reduce disability due to common mental health problems such as anxiety, depression, and comorbid substance misuse. As public awareness increases, the demand for mental health care far outstrips the capacity of health systems to provide access to quality care [
For young people, there is typically still a prolonged delay between the onset of first symptoms and initial treatment contact [
When considering how to provide such services at scale, eHealth (electonic health is a relatively recent health care practice that is supported by internet and/or other technologies) [
While there are concerns about potential lack of accountability in the mHealth field and that online communications could miss nonverbal cues that can ultimately impact empathy and patient satisfaction [
Imported from general medicine, the concept and application of clinical staging to mental health disorders seeks to redefine traditional diagnostic systems by placing individuals on a spectrum from early identification of nonspecific or mixed forms of mental symptoms through to more discrete disorders and then recurrent and persistent forms of illness [
As the staging framework recognizes the continuum of illness progression, it also encourages more personalized and responsive care at each point of the spectrum [
The process for determining stages has been outlined previously by Cross and colleagues [
Stage 0 : no symptoms; person at risk of disorder
Stage 1a: help-seeking; person with mild symptoms and mild functional impacts
Stage 1b: attenuated syndrome; person with mixed or ambiguous symptoms and moderate to severe functional impacts
Stage 2 : discrete disorders such as clear episodes of psychotic, manic, or severe depressive disorders
Stage 3 : recurrent or persistent disorder
Stage 4 : severe, persistent, and unremitting illness
Second, the assessment offers the possibility of immediate recommendations, support, and interventions anytime, anywhere, through a personalized dashboard of results (an easy-to-read clinical report with infographics) upon completion of the online assessment. Third, the assessment breaks down traditional geographical and socioeconomic barriers by increasing access to any care but specifically to more specialized assessment.
The aim of this study was to test new and innovative technologies to assess clinical stage in youth-specific mental health services using a prototypic online system known as the Mental Health eClinic (MHeC) [
Participants were recruited from 2 youth-specific mental health services (
Within these
The University of Sydney’s Human Research Ethics Committee approved the study (protocol number 2014/689). All participants were provided with information about the study prior to participating and consenting. Parental consent was also obtained for participants aged 16 and 17 years.
In order to test the online assessment within the MHeC, all eligible participants were invited to complete both the online assessment and standard assessment in face-to-face services. Participants were randomly allocated and counterbalanced by a 1-to-3 ratio to either undertake the face-to-face assessment or online assessment first. Considering the online assessment was a new method of assessment, an unequal randomization was preferred in order to minimize the impact of learning effects [
The face-to-face assessment included completion of the
The online assessment was based on the staging model as developed and adapted for early intervention youth mental health services. This assessment included 3 components (
Domain 1: home and environment
Domain 2: education and/or employment
Domain 3: activities
Domain 4: drugs and alcohol
Domain 5: relationships and sexuality
Domain 6: conduct difficulties and risk-taking
Domain 7: anxiety
Domain 8: eating
Domain 9: depression and suicide
Domain 10: psychosis and mania
Face-to-face consultation with a clinician
Self-report Web-based survey
Module 1: collects demographic information
Module 2: assesses medical history
Module 3: screens for prevalent mental health conditions [
Module 4:
Screens for hypomanic symptoms (items derived from the Altman Self-Rating Mania Scale [
Screens for psychotic symptoms (items derived from the Community Assessment of Psychotic Experiences Positive Symptoms Scale [
Measures psychological distress with the 10-item Kessler Psychological Distress Scale [
Measures somatic distress with the Somatic and Psychological Health Report [
Module 5: Assesses self-harm behaviors and suicidality using the Suicidal Ideation Attributes Scale [
Module 6: Assesses tobacco, alcohol and substance use—items derived from the Alcohol Use Disorders Identification Test [
Module 7: Measures physical activity using the International Physical Activity Questionnaire [
Module 8: Assesses sleep behaviors using 4 sleep-related items from the Quick Inventory of Depressive Symptomatology [
Module 9: Assesses eating behaviors with items derived from the Eating Disorder Examination [
Module 10: Measures social connectedness—items derived from the Perceived Social Support/Conflict Measure [
Immediate dashboard of results
Video visit with a clinician
The video visit of the online assessment included a brief, semistructured interview (
At the conclusion of each face-to-face session and video visit, all clinicians determined stage. These results were then collapsed into 2 groups: stage 1a or stage 1b and above (stage 1b+). Participants in stage 1a were help-seeking with mild symptoms and mild functional impairment while those in stage 1b+ were experiencing more severe symptoms and functional impairment. This key differentiation is predictive of clinical course and can be used to allocate service resources preferentially to those in greater need. Clinicians also completed the Social and Occupational Functioning Assessment Scale (SOFAS), which measures an individual’s functional status not directly related to the severity of their psychological symptoms [
In order to validate online assessment and staging classification, 2 trained clinicians (LOP and AT) were present during all video visits until such time their interrater agreement was considered reliable; that is, LOP (rater A) and AT (rater B) conducted alternating video visits while the other was present but not in view of the webcam (as per ethics approval and consent obtained from the young person). Raters A and B then determined stage independently, and once substantial concordance was sufficiently reached, LOP and AT conducted any remaining video visits with or without the other present.
All statistical analyses were performed using SPSS Statistics for Mac 22.0 (IBM Corp). Group differences in demographic, functional, and clinical variables were assessed using nonparametric Kruskal–Wallis test (
Interrater analyses determined degree of agreement of staging results between face-to-face and online clinicians as well as the 2 individual online clinicians (ie, rater A vs rater B). Cohen kappa statistic [
A total of 204 young people were identified as eligible to participate in the study. Based on a 1-to-3 random allocation counterbalancing ratio, 54 participants were invited to undertake standard face-to-face assessment first and 150 participants were invited to undertake the online assessment first; 125 participants were from
As shown in
The mean age of all participants was 20.35 (SD 2.63, range 16 to 25) years, 71% (51/72) were female, and 51% (37/72) had completed or partially completed tertiary education. Participants reported moderate distress levels (10-item Kessler Psychological Distress Scale mean 28.93, SD 8.42, range 10 to 50) with almost three-quarters (53/72, 74%) of the sample currently experiencing anxious and/or depressive symptoms. Nearly one-third (21/72, 29%) of participants screened positive for hypomanic symptoms, and one-third (24/72, 33%) screened positive for psychotic-like symptoms.
Almost half (35/72, 49%) of participants reported self-harm. Using our digitally smart Suicidality Escalation Protocol [
In order to validate the online assessment and staging classification, the trained clinicians were both present in 14 video visits until agreement was measured as substantial (kappa=.76,
Interrater agreement between online rater A and online rater B by assignment of clinical stage.
Online rater A | ||
Online rater B | Stage 1a (n=16), n (%) | Stage 1b+ (n=32), n (%) |
Stage 1a (n=17), n (%) | 14 (29) | 3 (6) |
Stage 1b+ (n=31), n (%) | 2 (4) | 29 (61) |
Interrater agreement between face-to-face and online clinicians by allocation to clinical stage.
Online assessment | ||
Face-to-face clinical assessment | Stage 1a (n=27), n (%) | Stage 1b+ (n=45), n (%) |
Stage 1a (n=42), n (%) | 23 (32) | 19 (26) |
Stage 1b+ (n=30), n (%) | 4 (6) | 26 (36) |
To calculate interrater agreement for assigning stage, participants were entered into a 2-by-2 comparison of stage assigned (stage 1a vs stage 1b+) and type of clinician who assigned that stage: face-to-face clinician versus online clinician (
Comparing stage 1b+ (disagree) with those participants determined as stage 1a (agree) by both clinician types, post hoc analyses showed that almost all young people in stage 1b+ (disagree) reported a previous history of mental health problems (χ21=10.71,
There was also a major discrepancy between the face-to-face and the online clinicians in categorizing the symptom severity for the participants allocated to stage 1b+ (disagree) group; face-to-face clinicians considered this group as normal (not at all ill) whereas online clinicians assigned a mildly ill classification. Among the online observations, the symptomatology of this group was considered to be significantly more pronounced compared to the stage 1a (agree) participants (CGI-S median rating of borderline ill;
When comparing stage 1b+ (disagree) with those in stage 1b+ (agree), post hoc analysis showed that participants assessed as stage 1b+ (agree) had significantly higher levels of suicidal ideation on the SIDAS (
Post hoc analysis with stage 1a (disagree) (stage 1a by online clinicians but assessed as stage 1b+ by face-to-face clinicians) participants was not conducted due to insufficient cell size.
Median scores and significance test results for self-reported variables among groups.
Characteristics | Stage 1a (agree)a (n=23), n (%) | Stage 1b+ (agree)b (n=26), n (%) | Stage 1b+ (disagree)c (n=19), n (%) | Significance test |
Post hoc |
|||
a vs c | b vs c | |||||||
Female, n (%) | 15 (65) | 18 (69) | 14 (74) | 1.70e (.72) | —f | — | ||
Age in years, median (IQR)g | 20.00 (4) | 20.50 (4) | 21.00 (4) | 0.78d (.86) | — | — | ||
1.58e (.71) | ||||||||
Secondary, n (%) | 12 (52) | 14 (54) | 8 (42) | — | — | — | ||
Tertiary, n (%) | 11 (48) | 12 (46) | 11 (58) | — | — | — | ||
K-10h, median (IQR) | 25 (13) | 32.0 (9) | 28.0 (13) | 5.51d (.14) | — | — | ||
Depression/anxiety (current), n (%) | 16 (70) | 22 (85) | 14 (74) | 6.03e (.09) | — | — | ||
Hypomanic-like issue (current), n (%) | 5 (22) | 10 (38) | 6 (32) | 2.91e (.38) | — | — | ||
Psychotic-like issue (current), n (%) | 5 (22) | 12 (46) | 7 (37) | 4.92e (.15) | — | — | ||
Mental health history, n (%) | 11 (48) | 20 (77) | 18 (95) | 11.83e (.005) | .001 | .21 | ||
Lifetime self-harm, n (%) | 7 (30) | 20 (77) | 7 (37) | 13.28e (.003) | .67 | .007 | ||
SIDASi, median (IQR) | 1 (4) | 9.5 (24) | 1 (5) | 12.59d (.006) | .71 | .008 | ||
Suicide planning history, n (%) | 0 (0) | 12 (46) | 7 (37) | 17.75e (<.001) | .002 | .53 | ||
Suicide attempt history, n (%) | 0 (0) | 6 (23) | 1 (5) | 6.98e (.04) | .45 | .21 | ||
Lifetime substance misuse, n (%) | 17 (74) | 18 (69) | 14 (74) | 0.36e (.98) | — | — | ||
Cannabis weekly, n (%) | 1 (4) | 8 (31) | 6 (32) | 7.60e (.04) | .03 | .95 | ||
Substances to cope with emotions, n (%) | 6 (26) | 18 (69) | 8 (42) | 10.85e (.009) | .24 | .07 |
aStage 1a by online and face-to-face clinicians.
bStage 1b+ by online and face-to-face clinicians.
cStage 1b+ by online clinicians but assessed as Stage 1a by face-to-face clinicians.
dKruskal–Wallis test, 2-tailed.
eFET: Fisher exact test, 2-tailed.
fNot applicable.
gIQR: Interquartile range.
hK-10: 10-item Kessler Psychological Distress Scale.
iSIDAS: Suicidal Ideation Attributes Scale.
Median scores and significance test results for clinician-reported variables among groups.
Tests | Stage 1a (agree)a (n=23), n (%) | Stage 1b+ (agree)b (n=26), n (%) | Stage 1b+ (disagree)c (n=19), n (%) | Significance test |
Post hoc |
||||||||
a vs c | b vs c | ||||||||||||
Face-to-face, median (IQR)f | 2.0 (1) | 3.0 (2) | 1.0 (1) | 37.04 (<.001) | .83 | <.001 | |||||||
Online, median (IQR) | 2.0 (1) | 4.0 (1) | 3.0 (1) | 35.29 (<.001) | <.001 | .01 | |||||||
Face-to-face, median (IQR) | 75.0 (9) | 69.0 (15) | 75.0 (5) | 12.17 (.007) | .10 | .08 | |||||||
Online, median (IQR) | 75.0 (9) | 60.0 (10) | 71.0 (10) | 25.33 (<.001) | .01 | .003 |
aStage 1a by online and face-to-face clinicians.
bStage 1b+ by online and face-to-face clinicians.
cStage 1b+ by online clinicians but assessed as stage 1a by face-to-face clinicians.
dKruskal–Wallis test, 2-tailed.
eCGI-S: Clinical Global Impression Scale–Severity.
fIQR: Interquartile range.
gSOFAS: Social and Occupational Functioning Assessment Scale.
The MHeC presents a new and innovative method for determining key clinical service parameters. While there was fair agreement between the staging classifications after both online and face-to-face assessment in the majority of cases (68%, kappa=.39), an important area of difference did emerge. During face-to-face assessments, clinicians tended to rate stage more conservatively compared to clinicians acting with the assistance of the MHeC.
Among the discordant cases, in 26% of cases face-to-face assessment appeared to place less emphasis on lifetime history of mental health problems. By contrast, the online assessment placed greater focus on past history of mental health problems
There are a range of possible explanations for this important difference between the face-to-face and online assessments, including (1) face-to-face assessment places greater emphasis on current symptomatology, (2) online clinicians made specific use of more extensive data collection about past as well as current symptomatology that was collected prior to the video visit (and as a consequence, their clinical assessment used all available data relevant to assign stage), and/or (3) face-to-face clinicians may be more influenced by the consequences of their clinical assessment for allocating service resources—that is, higher stage ratings are reserved preferentially for those who are perceived to be in need of more intensive or prolonged care.
Assessing the mental health of young people and their need for immediate or ongoing health care is a real challenge for clinicians and youth mental health services. Specifically, this includes being able to distinguish normative emotional development and brief stress-related responses from emerging mental disorders [
By entering information online, young people can complete a self-report Web-based survey in their own time whenever and wherever they prefer. This provides greater choice at the forefront of mental health care by directly and immediately responding to young people’s needs [
As community-based and outpatient mental health care is limited, all services struggle with high demand pressures [
A systematic clinician bias toward underrating young people to stage 1a could have deleterious effects on service users. Our previous research has shown that 15% of people in stage 1b transition to stage 2 within 1 year [
Finally, one of the most obvious advantages of the online assessment addresses geographical barriers. In this trial, 10% (8/82) of the video visits were completed with 1 of our clinicians online while she was overseas (LOP traveled overseas due to work commitments) using secure videoconference software. This positions online assessment as an efficient solution connecting young people not only with care but with the right clinician regardless of their location, potentially saving time and money for young people, clinicians, and services.
One limitation of this study is the sample size because the face-to-face arm suffered from greater participant attrition. It is possible that participants who had already completed the face-to-face assessment felt that completing a second assessment online was an unnecessary use of time. Additionally, this study required people to complete all of the 4 main components (tablet questionnaire, face-to-face interview, Web-based survey, and video visit) within 2 weeks of the first interview, and the majority of attrition in both study arms was accounted for by this stringent protocol. Although the unequal randomization (1-to-3) favored the analysis with the reduction of the impacts of the learning curve, it compromised the power of the study. Future research is needed with a 1-to-1 randomization, increasing the power of the comparison.
Our study revealed poor interrater reliability on CGI-S allocation between face-to-face and online clinicians. There are 2 possible explanations for this disagreement. Face-to-face clinicians do not use the CGI-S in their daily practice and therefore are less familiar with its application, while online clinicians had used this tool in other research studies and were consequently more familiar with its application. Additionally, due to the CGI-S’s instruction (“Considering your total clinical experience with this particular population, how mentally ill is the patient at this time?”), it has been acknowledged that clinician experience could explain the variability in the CGI-S scoring [
Future research is needed to evaluate the engagement, efficacy, and effectiveness of MHeC’s online assessment within real-world service environments. It would also include formal validation of the online assessment against gold standard assessment and testing the effectiveness of any education and training program that might be developed to supplement these new and innovative technological solutions for the delivery of better mental health care.
This study highlights the use of new and innovative technologies to assess clinical stage in early intervention youth mental health services through an online MHeC. It promotes systematic assessment of lifetime severity and complexity of clinical presentations while concurrently addressing risk assessment in a shorter period of time. The MHeC has the potential to be adapted to varied settings in which young people are connecting with traditional clinical services and assist in providing the right care at the right time.
Video visit semistructured interview.
Consolidated Standards of Reporting Trials diagram indicating the flow of participants through the study.
Clinical Global Impression Scale-Severity
Fisher exact test
Kruskal–Willis test
Home, Education and Employment, Eating, Activities, Drugs and Alcohol, Sexuality, Suicide and Depression, Safety
intraclass correlation coefficient
interquartile range
Mental Health eClinic
Suicidal Ideation Attributes Scale
Social and Occupational Functioning Assessment Scale
Mann–Whitney test
We would like to thank all the participants in the study from
IBH was an inaugural commissioner on Australia’s National Mental Health Commission (2012-2018). He is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC), University of Sydney. The BMC operates an early-intervention youth service at Camperdown under contract to