Background: Mental health difficulties in young people are increasing, and there is a need for evidence on the effectiveness of digital interventions to increase opportunities for supporting mental health in young people. Such studies are complicated due to issues of implementation and adoption, outcome measurement, and appropriate study designs.
Objective: The objective of this study was to examine the effectiveness of an mHealth intervention (ReZone) in reducing mental health difficulties in young people.
Methods: The cluster-randomized controlled trial enrolled 409 participants aged 10-15 years, and classes were allocated to ReZone or management as usual. Self-reported questionnaires were completed at baseline and 3-month follow-up.
Results: There were no significant differences between the ReZone condition and management as usual in the self-reported outcome measures. However, there were 3467 usage sessions, which corresponds to 16.9 times per student (total of 205 students) in classes allocated to ReZone.
Conclusions: It is essential to publish studies that do not show significant differences, as these findings can still contribute to the literature, help in learning, and inform the direction of future work. The results reported in this paper could be due to a range of reasons, including whether ReZone has the scope to impact change or limitations related to the setting, context, and appropriateness of an RCT. The findings of this study suggest that ReZone was implemented and adopted.
International Registered Report Identifier (IRRID): RR2-10.2196/resprot.7019
Mental health difficulties are increasing among young people. Emotional disorders are the most prevalent and show the most significant increase over time and with age [, ]. Recent figures show that 1 in 8 (12.8%) people aged 5-19 years had at least one mental disorder when assessed [ ]. Mobile health (or mHealth) interventions have the potential to provide mental health support at scale, and there is a need for evidence as to the effectiveness of digital interventions for supporting mental health in young people [ ]. However, non-adoption and implementation issues present substantial barriers to effectively doing so [ ]. Additionally, questions about whether randomized controlled trials are always the most appropriate method of evaluation need to be considered in the context of practicality and feasibility factors [ - ].
The authors have co-designed an mHealth intervention, ReZone, with young people, parents/carers, and teachers, based on cognitive behavioral therapy (CBT), mindfulness, and attention bias modification training (ABMT). This short paper will report the findings from a cluster randomized controlled trial (RCT) to test the hypothesis that young people in classes receiving ReZone have fewer behavioral difficulties than young people in classes receiving management as usual. Full details on the methods and development are reported in the protocol  and implementation paper [ ].
Participants and Procedure
A total of 49 classes were recruited (27 mainstream, 22 alternative provision), resulting in a sample of 409 students (mean age 10.9, SD 1.44 years; 56% male;). Schools were recruited via newsletters and events published in our Schools in Mind network. Participants were enrolled from 12 mainstream primary schools (307 students) and 11 secondary alternative provision schools (103 students). A total of 66 people were lost to follow-up, resulting in an attrition rate of 16%. All students (aged 10-15 years) in schools taking part in the project were eligible to participate. Classes were randomized using a true random number generator in blocks of two by an independent trials unit, to management as usual or ReZone stratified by school type (alternative provision vs. mainstream) after classes were recruited and baseline measures assessed. Ethical approval was received from University College London (UCL) Research Ethics Committee (number: 7969/001). The present research is reported in line with CONSORT guidelines [ ].
|Demographics||Overall||Mainstream primary||Alternative provision|
|Control||Intervention||P value||Control||Intervention||P value|
|Age (years), mean (SD)||10.9 (1.44)||10.25 (0.43)||10.28 (0.45)||.53||13.6 (1.48)||12.3 (1.45)||<.001|
|Male, n (%)||227 (56)||69 (43.9)||75 (50.3)||.22||38 (80.9)||45 (80.4)||.90|
|Ethnicity, n (%)||.74||.90|
|White||247 (60)||98 (62.4)||82 (55.0)||31 (66.0)||36 (64.3)|
|Mixed-race||29 (7)||N/Aa||6 (4.0)||7 (14.9)||9 (16.1)|
|Asian||78 (19)||34 (21.7)||40 (26.9)||3 (6.4)||1 (1.8)|
|Black||40 (10)||11 (7.0)||15 (10.1)||5 (10.6)||9 (16.1)|
|Other||15 (4)||5 (3.2)||5 (3.4)||7 (14.9)||1 (1.8)|
|Disabled, n (%)||28 (3.4)||2 (1.3)||2 (1.3)||.99||6 (13.0)||4 (7.3)||.28|
|Alternate provision schools, n (%)||103 (25)||N/A||N/A||N/A||N/A||N/A||N/A|
|English first language, n (%)||352 (87)||N/A||N/A||N/A||N/A||N/A||N/A|
aN/A: not applicable.
Intervention: ReZone App
Full details of the ReZone intervention are described in the protocol . The intervention primarily aims to reduce behavioral difficulties. It is available as a web-based smartphone and tablet application flexibly designed to be used in groups or individually, with or without adult support depending on the user’s capability and confidence. This flexibility also extends to the order and frequency in which elements of the tool are used. However, it is required to have an initial guided session or tutorial with an adult. This flexible approach to implementation aimed to fit within a variety of schools and their differing student characteristics, priorities, and timetables.
The primary features of ReZone () are presented below.
The primary features of ReZone.
Stress bucket. The stress bucket lets the user add any stressors that they are experiencing to a bucket. They are then able to introduce activities that help them cope with each stressor. They can see the water in the bucket rise and fall as they add and relieve stressors. If the bucket reaches 50 stress points, it will overflow.
Timeout. Timeout asks the user to think through a time when they have felt stressed, angry, or upset. The user works through the events leading up to feeling this way and what happened afterward. The user can also think through what they could have done differently, thus guiding them to develop a behavioral plan. The timeout visualization is a rocket, and each thought process creates an exhaust cloud.
Chill out. Chill out uses breathing to help the user calm down and relax. Each chill out activity is based around an object or animal (ie, rabbit, jellyfish, ball, or square), using breathing in different ways.
Art therapy. The user can choose between a castle, dinosaur, fish, goat, heart, helicopter, unicorn, rocket, footballer, sea, or turtle to color in. There is a range of colors and tools to complete the drawing.
Happy faces. The user is given 30 seconds to find as many happy faces as they can, amongst other faces depicting negative emotions.
Game. In the game Balloon Blast, the user taps the screen to move a balloon up, trying to avoid all the obstacles as hitting one will mean the game is over. The game provides a break or reward for the user in between uses of the other features.
All measures were self-reported by young people and administered in school by researchers or school staff in paper form. Baseline demographic data were collected and are reported in. Emotional (α=.78) and behavioral difficulties (α=.81) were measured using the 16-item Me and My School (M&MS) questionnaire [ ]. Mental wellbeing was measured using the 7-item Short Warwick-Edinburgh Mental Wellbeing Scale (α=.83) [ ]. Empowerment was measured using the 6-item “self” subscale of the Youth Empowerment Scale-Mental Health (α=.73) [ ]. Health-related quality of life was measured using the 6-item (with global health visual analog scale) EQ-5D-Y (α=.63) [ ].
Intervention usage data were collected in ReZone and exported to the data analysis software. There were 3467 usage sessions, which corresponds to 16.9 times per each of the 205 students in classes allocated to ReZone. The game (n=752), breathing exercises (n=811), and happy faces (n=1169) were used the most; stress bucket (n=404), art (n=225), and time out (n=106) were used the least. Based on teacher consultations, which are discussed elsewhere , usage methods and frequency varied across schools and students due to a variety of factors, including student characteristics, teacher priorities, and school environment.
Means and standard deviations, for all measures, across conditions at baseline and follow-up, were calculated. Intention-to-treat and per-protocol analyses with four multilevel regressions were tested for each of the outcome variables with time nested within students within classrooms. Per-protocol analyses included only those that had completed follow-up. Model 0 was computed as the null model to examine the change in outcome over time, without predictors. The intraclass correlation coefficients for each measure were 17% (emotional difficulties), 24% (behavioral difficulties), 7% (empowerment), 9% (wellbeing), and 3% (quality of life). We then entered school type (ie, alternative provision vs mainstream) as a fixed effect predictor to Model 1. In Model 2, patient-level grand mean-centered age, ethnicity, gender, and disability were entered as predictors. Demographics were controlled for in the analyses as there were significant differences in gender between groups at baseline within mainstream primary schools and age between groups within alternative provision schools (). In the Final Model (Model 3), condition (ReZone vs management as usual) was entered as a predictor. The likelihood ratio test was used to compare the fit of Models 1, 2, and 3. For example, Model 1 was significant for behavioral difficulties and therefore compared to Model 2, which was significant. Models 1 and 2 were therefore retained in the final model, with Model 3 (final model) being compared to Model 2 using the likelihood ratio test. Models 1 and 2 were significant for empowerment and wellbeing and therefore retained in the final model. For emotional difficulties and quality of life, only Model 2 was significant and retained in the final Model.
The means and standard deviations of all study measures at baseline and postintervention are presented in. The likelihood ratio test for the intention to treat analysis was not significant for any of the final models compared to the previous significant model. There were no significant differences between the ReZone condition and management as usual in emotional difficulties (χ21=0, P>.05), behavioural difficulties (χ21=0.01, P>.05), empowerment (χ21=0.01, P>.05), wellbeing (χ21=0.05, P>.05), or quality of life (χ21=0.3154, P>.05). The per-protocol results showed similar patterns.
|Mainstream primary||Alternative provision|
|Outcome||Intervention – T1a||Intervention – T2b||Control – T1||Control – T2||Intervention – T1||Intervention – T2||Control – T1||Control – T2|
|Emotional difficulties||5.56 (3.38)||5.18 (3.74)||6.27 (3.38)||5.43 (3.27)||6.08 (3.91)||5.61 (4.12)||5.48 (3.77)||6.08 (4.03)|
|Behavioral difficulties||3.31 (2.27)||3.28 (2.44)||3.32 (2.24)||3.18 (2.04)||5.45 (2.37)||5.11 (2.34)||5.57 (2.35)||5.82 (2.52)|
|Empowerment||24.53 (4.57)||25.69 (4.25)||24.65 (4.09)||25.13 (4.03)||22.96 (4.96)||23.61 (5.74)||23.35 (4.02)||23.79 (4.25)|
|Wellbeing||24.11 (4.78)||24.94 (5.41)||24.41 (5.18)||24.94 (5.06)||21.09 (5.90)||23.61 (5.74)||21.76 (3.98)||21.93 (4.65)|
|Quality of life||6.13 (1.60)||6.46 (5.86)||6.28 (1.48)||6.29 (6.06)||6.20 (1.24)||4.04 (3.08)||6.30 (1.32)||2.91 (3.14)|
aT1: Time 1.
bT2: Time 2.
This cluster randomized controlled trial examined the effectiveness of ReZone in reducing behavioral and mental health difficulties in young people. The study did not show a significant difference between those using ReZone and management as usual in the reduction of behavioral and emotional difficulties and improving empowerment, wellbeing, and quality of life.
Potential limitations of the study, implementation, and adoption barriers to digital interventions within schools have been discussed in depth . Implementation barriers to new digital interventions in schools include the technology itself, the adopter system, and the organization [ ]. The process of embedding ReZone into the school system can sometimes leave minimal time to utilize it successfully. Implementation strategies differed, with varying levels of teacher encouragement and consistency of scheduled ReZone use. Competing teacher priorities, opinions, and seniority of the school advocate could have impacted the time and drive to use the tool [ ].
ReZone was used a total of 3467 times throughout the 3-month study period, and current usage data show that ReZone is still being used 10 months poststudy (501 sessions over the last 3 months). Aggregated objective usage data is useful to indicate overall acceptability and use. However, teacher consultations (fully reported in ) suggested students either used ReZone frequently or not at all. Analysis of a dose-response effect could be improved by gathering further quantifiable individual school usage information related to implementation.
The usage statistics and teacher consultations indicated happy faces and breathing were the most used activities, while the stress bucket, art, and time out were used the least. Happy faces and breathing can be completed in only 1-2 minutes; they are a fun challenge/game type of activity, and most of the time can be completed without teacher assistance. The stress bucket and time out require more time to work through thought processes, requiring the user to be willing to think and talk about their feelings, needing teacher support (which is not always feasible).
The challenges discussed link to the emerging concept of gamification within digital mental health tools [, ]. There are some basic elements of gamification within the most popular elements of ReZone. Further work on gamification combined with other crucial processes such as co-design, could help address the broader issue of nonadoption and sustainability, with many users reporting that they stop the use of a health app after only two weeks [ ]. This lack of consistency could be particularly applicable when working with young people in such settings as alternative provision schools. Pupil behavior often places higher demands on staff in alternative provision schools than in mainstream schools, and disengagement and loss of concentration are common issues.
Measures used were related to overall mental health, wellbeing, and behavior leading us to question how often a tool such as ReZone would need to be used to effect changes in broad domains and specifically in schools. If tools such as ReZone do not provide large-scale, substantial longer-term changes, we must also ask whether there is any impact on immediate feelings, behavior, and interactions. We did attempt to measure this by asking teachers to complete a few short questions related to how well students refocused and re-engaged after using ReZone. However, due to practical issues such as teacher demand, these were not completed. On reflection, this could have also been something students were asked to complete. It is also possible that effects were delayed, and further long-term data collection, such as one-year post-trial, may have shown differences [- ].
Due to the complex nature of digital interventions, traditional RCTs may be insufficient to assess efficacy . As this study shows, there are multiple objective and subjective data sources available, including quantitative self-report data, objective usage data, and subjective qualitative data, while digital and school studies present particular challenges in adherence, sustainability, and roll-out [ ]. As with psychotherapy research in general and a reliance on self-report measures, participant blinding was not possible, although true and independent randomization was used. Additionally, due to this being a pragmatic trial, there may have been other interventions the school and participants may have been using.
It is essential to publish negative results to avoid publication bias. These studies can still make a substantial contribution to the literature, learning, and informing the direction of future work . The negative results of this study could indicate that ReZone has a limited scope to impact change, that its utility is hampered by issues with implementation and nonadoption, and limitations of mHealth research in a school context. The results may also suggest the limited applicability of the RCT study design for evaluating tools such as ReZone.
The research was supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Bart’s Health NHS Trust. The views expressed are those of the authors and not necessarily those of the NHS, NIHR, or Department of Health.
Conflicts of Interest
- Deighton J, Lereya ST, Casey P, Patalay P, Humphrey N, Wolpert M. Prevalence of mental health problems in schools: poverty and other risk factors among 28,000 adolescents in England. Br J Psychiatry 2019 Jan 30:1-3. [CrossRef] [Medline]
- Fink E, Patalay P, Sharpe H, Holley S, Deighton J, Wolpert M. Mental Health Difficulties in Early Adolescence: A Comparison of Two Cross-Sectional Studies in England From 2009 to 2014. J Adolesc Health 2015 May;56(5):502-507. [CrossRef] [Medline]
- Mental Health of Children Young People in England, 2017. London: NHS Digital; 2018. URL: https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/2017/2017 [accessed 2020-06-19]
- Hollis C, Falconer CJ, Martin JL, Whittington C, Stockton S, Glazebrook C, et al. Annual Research Review: Digital health interventions for children and young people with mental health problems - a systematic and meta-review. J Child Psychol Psychiatry 2017 Apr 10;58(4):474-503. [CrossRef] [Medline]
- Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies. J Med Internet Res 2017 Nov 01;19(11):e367 [FREE Full text] [CrossRef] [Medline]
- Donner A, Klar N. Pitfalls of and controversies in cluster randomization trials. Am J Public Health 2004 Mar;94(3):416-422. [CrossRef] [Medline]
- Kaptchuk TJ. The double-blind, randomized, placebo-controlled trial. Journal of Clinical Epidemiology 2001 Jun;54(6):541-549. [CrossRef]
- Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop. J Med Internet Res 2017 Jun 29;19(6):e232 [FREE Full text] [CrossRef] [Medline]
- Edbrooke-Childs J, Smith J, Rees J, Edridge C, Calderon A, Saunders F, et al. An App to Help Young People Self-Manage When Feeling Overwhelmed (ReZone): Protocol of a Cluster Randomized Controlled Trial. JMIR Res Protoc 2017 Nov 03;6(11):e213. [CrossRef]
- Edridge C, Deighton J, Wolpert M, Edbrooke-Childs J. The Implementation of an mHealth Intervention (ReZone) for the Self-Management of Overwhelming Feelings Among Young People. JMIR Form Res 2019 May 02;3(2):e11958 [FREE Full text] [CrossRef] [Medline]
- Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med 2010 Mar 24;8(1):18 [FREE Full text] [CrossRef] [Medline]
- Patalay P, Deighton J, Fonagy P, Vostanis P, Wolpert M. Clinical validity of the Me and My School questionnaire: a self-report mental health measure for children and adolescents. Child Adolesc Psychiatry Ment Health 2014;8(1):17 [FREE Full text] [CrossRef] [Medline]
- Stewart-Brown S, Tennant A, Tennant R, Platt S, Parkinson J, Weich S. Internal construct validity of the Warwick-Edinburgh Mental Well-being Scale (WEMWBS): a Rasch analysis using data from the Scottish Health Education Population Survey. Health Qual Life Outcomes 2009 Feb 19;7(1):15 [FREE Full text] [CrossRef] [Medline]
- Walker JS, Thorne EK, Powers LE, Gaonkar R. Development of a Scale to Measure the Empowerment of Youth Consumers of Mental Health Services. Journal of Emotional and Behavioral Disorders 2009 Jun 05;18(1):51-59. [CrossRef]
- EuroQol Group. EQ-5D-Y user guide: basic information on how to use the EQ-5D-Y instrument. 2014. URL: https://euroqol.org/wp-content/uploads/2016/09/EQ-5D-Y_User_Guide_v1.0_2014.pdf [accessed 2017-01-01]
- Johnson D, Deterding S, Kuhn K, Staneva A, Stoyanov S, Hides L. Gamification for health and wellbeing: A systematic review of the literature. Internet Interv 2016 Nov;6:89-106 [FREE Full text] [CrossRef] [Medline]
- Pramana G, Parmanto B, Lomas J, Lindhiem O, Kendall PC, Silk J. Using Mobile Health Gamification to Facilitate Cognitive Behavioral Therapy Skills Practice in Child Anxiety Treatment: Open Clinical Trial. JMIR Serious Games 2018 May 10;6(2):e9 [FREE Full text] [CrossRef] [Medline]
- Dorsey ER, Yvonne Chan Y, McConnell MV, Shaw SY, Trister AD, Friend SH. The Use of Smartphones for Health Research. Academic Medicine 2017;92(2):157-160. [CrossRef]
- Bonell C, Allen E, Warren E, McGowan J, Bevilacqua L, Jamal F, et al. Effects of the Learning Together intervention on bullying and aggression in English secondary schools (INCLUSIVE): a cluster randomised controlled trial. The Lancet 2018 Dec;392(10163):2452-2464. [CrossRef]
- Foshee V, Bauman K, Greene W, Koch G, Linder G, MacDougall J. The Safe Dates program: 1-year follow-up results. Am J Public Health 2000 Oct;90(10):1619-1622. [CrossRef] [Medline]
- Foshee VA, Bauman KE, Ennett ST, Linder GF, Benefield T, Suchindran C. Assessing the long-term effects of the Safe Dates program and a booster in preventing and reducing adolescent dating violence victimization and perpetration. Am J Public Health 2004 Apr;94(4):619-624. [CrossRef] [Medline]
- Franco A, Malhotra N, Simonovits G. Social science. Publication bias in the social sciences: unlocking the file drawer. Science 2014 Sep 19;345(6203):1502-1505 [FREE Full text] [CrossRef] [Medline]
|ABMT: attention bias modification training|
|CBT: cognitive behavioral therapy|
|RCT: randomized controlled trial|
Edited by G Eysenbach; submitted 08.04.19; peer-reviewed by R Meiksin, B Yanez; comments to author 06.07.19; revised version received 31.10.19; accepted 23.03.20; published 27.07.20Copyright
©Chloe Edridge, Miranda Wolpert, Jessica Deighton, Julian Edbrooke-Childs. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 27.07.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.