Background: The Online Remote Behavioral Intervention for Tics (ORBIT) study was a multicenter randomized controlled trial of a complex intervention that consisted of a web-based behavioral intervention for children and young people with tic disorders. In the first part of a two-stage process evaluation, we conducted a mixed methods study exploring the reach, dose, and fidelity of the intervention and contextual factors influencing engagement.
Objective: This study aims to explore the fidelity of delivery and contextual factors underpinning the ORBIT trial.
Methods: Baseline study data and intervention usage metrics from participants in the intervention arm were used as quantitative implementation data (N=112). The experiences of being in the intervention were explored through semistructured interviews with children (n=20) and parent participants (n=20), therapists (n=4), and referring clinicians (n=6). A principal component analysis was used to create a comprehensive, composite measure of children and young people’s engagement with the intervention. Engagement factor scores reflected relative uptake as assessed by a range of usage indices, including chapters accessed, number of pages visited, and number of log-ins. The engagement factor score was used as the dependent variable in a multiple linear regression analysis with various contextual variables as independent variables to assess if there were any significant predictors of engagement.
Results: The intervention was implemented with high fidelity, and participants deemed the intervention acceptable and satisfactory. The engagement was high, with child participants completing an average of 7.5 of 10 (SD 2.7) chapters, and 88.4% (99/112) of participants completed the minimum of the first four chapters—the predefined threshold effective dose. Compared with the total population of children with tic disorders, participants in the sample tended to have more educated parents and lived in more economically advantaged areas; however, socioeconomic factors were not related to engagement factor scores. Factors associated with higher engagement factor scores included participants enrolled at the London site versus the Nottingham site (P=.01), self-referred versus clinic referred (P=.04), higher parental engagement as evidenced by the number of parental chapters completed (n=111; ρ=0.73; P<.001), and more therapist time for parents (n=111; ρ=0.46; P<.001). A multiple linear regression indicated that parents’ chapter completion (β=.69; t110=10.18; P<.001) and therapist time for parents (β=.19; t110=2.95; P=.004) were the only significant independent predictors of child engagement factor scores.
Conclusions: Overall, the intervention had high fidelity of delivery and was evaluated positively by participants, although reach may have been constrained by the nature of the randomized controlled trial. Parental engagement and therapist time for parents were strong predictors of intervention implementation, which has important implications for designing and implementing digital therapeutic interventions in child and adolescent mental health services.
International Registered Report Identifier (IRRID): RR2-10.1186/s13063-019-3974-3
Tics are sudden, brief, rapid, and recurrent nonrhythmic movements or vocalizations that are more common in children and young people (CYP) than in adults  and more prevalent in males than in females [ ]. Tic onset typically occurs between the ages of 3 and 8 years (mean onset is 6 and 7 years of age) [ ], with the reported average age of greatest tic severity by the age of 10 years [ ]. Although most CYP with tics only require educational support as the main form of treatment [ ], there are interventions available for severe or disabling tics, as in Tourette syndrome or chronic tic disorders. Historically, pharmacotherapy, such as antipsychotics, has been the first line of treatment for severe tics; however, they often have undesirable side effects, such as weight gain and sleepiness [ ]. Behavioral interventions are appealing and effective alternatives to pharmacotherapy. However, they require the patient to invest time and energy in practicing demanding behavioral techniques, such as tic control or habit reversal. Despite the benefits and evidence-based effectiveness of behavioral therapies for tic disorders [ - ], there is great difficulty in patients accessing behavioral treatments because of a shortage of trained therapists [ ]. A promising development in increasing accessibility to behavioral treatments is the use of digital health interventions (DHIs) [ ]. Preliminary evidence suggests that DHIs are efficacious for CYP with tic disorders in pilot randomized controlled trials (RCTs) [ - ]. A study that has assessed DHIs for tic disorders is the Online Remote Behavioral Intervention for Tics (ORBIT) trial, which has been described in detail previously [ ] (see for a brief description).
Brief description of the ORBIT (Online Remote Behavioral Intervention for Tics) trial.
Design: A 10-week, 2-armed, parallel-group, single-blind randomized controlled trial with an embedded process evaluation.
Aim: To evaluate the effectiveness of a web-based, remote, therapist-supported, and parent-guided behavioral intervention for tics, initially developed and piloted in Sweden.
Intervention group: A total of 112 children and young people received 10 modules (called "chapters") of behavioral therapy following the principles of exposure and response prevention via a secure web-based platform, with access to a therapist, delivered over a period of 10-12 weeks.
Control group: A total of 112 children and young people received 10 chapters of psychoeducation via a secure web-based platform, with access to a therapist, delivered over a period of 10-12 weeks.
Primary outcome: Total Tic Severity Score on the Yale Global Tic Severity Scale at 3 months postrandomization.
Therapist role: Both children and parents had regular contact with a therapist during the 10-12 week period via messages that were sent within the treatment platform (resembling an email) or via telephone, if required. The therapist was also able to directly comment on exercises that the participant had been working on and give specific feedback to motivate participants.
Parent role: One or both of the child's parents received a separate log-in to the web-based treatment platform, where they could access their own chapters. The parent chapters contained information regarding parent coping strategies and how to support their child. The parents also had access to the assigned therapist.
The population impact of any given intervention depends on both its effectiveness and its reach, defined as the proportion of the target population who access the intervention . Although RCTs are the gold standard method for determining efficacy, additional data are needed before deciding whether an intervention should be adopted into mainstream health care. These additional data include understanding the reach of the intervention and the extent to which the data from an RCT, where the delivery of the intervention is often tightly controlled and monitored, can be extrapolated to use in routine health care. It has been argued that studies addressing questions about reach and effectiveness in routine care are needed [ , ]. However, such studies are expensive, and a process evaluation conducted alongside an RCT is an efficient method of maximizing the information yielded by the trial.
The Medical Research Council (MRC) has developed specific guidelines for conducting process evaluations of complex interventions . A complex intervention is defined as an intervention with several interacting components [ ], and the MRC outlines three essential components for evaluating complex interventions: implementation, mechanisms of impact, and context. Implementation can refer to how an intervention will be delivered within routine clinical practice, having shown efficacy in an outcome evaluation. However, this paper is concerned with another aspect of implementation: the extent to which the delivery of an intervention is achieved within the context of an RCT and the structures and processes through which an intervention is delivered as intended (ie, fidelity) [ ]. For complex interventions such as DHIs, an important component of implementation fidelity is the degree to which participants engage with the intervention and use it as intended. Effective engagement requires participants to register with the program and then continue to use it and apply the recommended behavioral techniques over time. The nonuse of DHIs is a well-recognized challenge (eg, the Eysenbach Law of Attrition [ ]) and can be considered in two parts: initial uptake (eg, registration and onboarding) and ongoing engagement.
To evaluate intervention implementation, the MRC guidelines for process evaluations suggest researchers assess (1) reach—the extent to which a target audience comes into contact with the intervention; (2) dose—how much of the intervention was delivered and received; (3) fidelity—the quality of what was delivered; and (4) adaptations—any modifications made to an intervention to achieve better contextual fit. The intended target audience for ORBIT was CYP with tic disorders; however, pertinent questions could be asked, such as whether there were socioeconomic biases in who was reached. In terms of dosage, the ORBIT protocol  states that the intervention should consist of 10 individual intervention chapters following a suggested frequency and a total duration of 10-12 weeks. The first four chapters delivered core content, including learning about tics and practicing suppressing tics. Completing these four chapters was designated as the minimum dose required for treatment completion by the ORBIT clinical team. There were six additional chapters offering reinforcement, further practice, and relapse prevention. For DHIs, the fidelity of delivery of the intervention is ensured by the web-based delivery platform. However, the intervention experienced by the user is highly dependent on the extent to which they engage with the intervention and use it as intended and the quality of delivery [ ]. Hence, in this paper, we look at usage and the proportion of participants receiving the predefined minimum effective dose of four or more chapters. Finally, understanding adaptations to the intended intervention involves exploring whether they improve the contextual fit or compromise the functioning of the intervention [ ] or whether they represent innovation or intervention drift [ ]. Participants were able to modify various intervention components, such as the tic stopwatch, which was used to self-time the length of tic control. This study aims to conduct the first part of a two-stage process evaluation of the ORBIT trial outlined in the study protocol [ ]. Part 1 focuses on intervention implementation by exploring the fidelity of delivery experienced by participants using usage statistics, reach, and acceptability of the intervention. It also investigates contextual factors associated with the observed variation in uptake and usage by examining the components specified in the MRC guidelines [ ]. Part 2 will focus on the impact mechanisms, and the findings will be reported in a future article. shows the two parts of the process evaluation: areas of research, explanatory data, and outcomes.
|Research questions||Process evaluation components||Explanatory data||Outcomes|
|Part 1. Intervention implementation (What is implemented and how?)|
|Part 2. Mechanisms of impact (How does it produce change?)|
|Part 1 and 2. Context (How do factors external to the intervention affect intervention implementation and change?)|
This study followed the MRC guidelines  for the process evaluation of complex interventions. It used a mixed methods, longitudinal design to explore the implementation fidelity of a web-based intervention for CYP with tics [ ] and the contextual factors that influenced the level of engagement.
The sample included in the quantitative phase of the process evaluation consisted of key information from all participants (N=112) from the intervention arm of the RCT. The sample included in the qualitative component of the process evaluation consisted of interviews with children and parent participants (target n≥20), interviews with all therapists delivering the intervention or supervising the therapists, and interviews with referring clinicians (target n>5).
Quantitative Data Collection
Quantitative process data were collected simultaneously with enrollment, intervention delivery, and outcome data collection in the main RCT.
Demographic and Clinical Data
Demographic and clinical information were recorded using a baseline demographic questionnaire. These data included the child’s age, residence (full postcode), gender, ethnicity, parental education level and occupation, all current suspected or confirmed diagnoses and interventions, and medication use.
Index of Multiple Deprivation
The index of multiple deprivation (IMD; 2019) is a relative measure of deprivation across seven different domains: income deprivation; employment deprivation; education, skills, and training deprivation; health deprivation and disability; crime; barriers to housing and services; and living environment deprivation . On the basis of the 6-digit zip code, the rank of deprivation associated with participants’ area of residence was calculated from 32,844 small areas or neighborhoods in England, with higher ranks indicating greater deprivation. Ranks were recoded into quintiles, with 1 being the most deprived and 5 being the least deprived.
Yale Global Tic Severity Scale
The primary outcome measure used in the ORBIT intervention was the Total Tic Severity Score (TTSS), as measured by the Yale Global Tic Severity Scale (YGTSS). The YGTSS is a valid and reliable clinician-rated scale , which scores the severity of motor and vocal tics separately by evaluating the number, frequency, intensity, complexity, and interference of tics. Each domain was scored on a 0-5 scale, and 2 tic severity scores were given: total motor (0-25) score and total vocal (0-25) score, which when combined give the TTSS (0-50).
Mood and Feelings Questionnaire
The Mood and Feelings Questionnaire (MFQ)  is a 33‐item measure evaluating depressive symptoms rated on a 3‐point scale: 0 is not true, 1 is sometimes, and 2 is true. Total scores range from 0 to 66, with higher scores reflecting more severe depression. A cutoff score of ≥29 is generally used to suggest clinically significant depression [ ].
Web usage data were collected and recorded from the participants throughout the trial. This included the following measures: number of chapters completed per child and per parent, total therapists’ time per child and per parent, individual therapist’s telephone time with participants, the volume of written communication (total number of characters) submitted by child and parent via the web-based system, total number of log-ins for child and parent, average time between each log-in (in days) for child and parent, and average pages visited per log-in for child and parent.
Satisfaction and Treatment Credibility
At the 3-week postrandomization point of treatment, all participants were asked to rate treatment credibility; two questions were asked: one relating to how well suited the participant felt the intervention was for helping CYP to manage their tics and the other question was about how much better they expected to feel as a result of the intervention. The responses were rated on a Likert scale of 0 to 4 for each question, with higher scores indicating higher treatment credibility. At the primary endpoint, all participants were asked to rate their satisfaction with the intervention. In total, eight satisfaction questions were asked with responses rated on a scale of 0-4, meaning that the overall satisfaction score was out of 32.
Qualitative Data Collection
Interviews with therapists and therapist supervisors involved in the ORBIT trial were conducted early in the study and near the end of recruitment to gain an understanding of their experience at different time points. Interviews with referring clinicians were conducted at the end of recruitment. Interviews with children participants and one of their parents were conducted following completion of the intervention at the 3-month (primary endpoint) follow-up assessment in the main RCT to minimize the risk of bias in the outcomes. Recruitment for the interviews began in August 2018 and ended in October 2019.
All interviews were conducted face-to-face, via telephone, or via videoconference (WebEx or Skype). Younger children were interviewed together with their parents, whereas older children (eg, aged >13 years) were interviewed separately. Participants were purposively sampled so that a diverse range of views on the intervention were voiced . This included ensuring that perspectives were heard from participants with a range of ages, gender, ethnicity, and level of interaction with the intervention. The overall sample enabled diverse intervention views and ensured that the data reached the saturation level [ ]. In addition to the interviews, at the end of the treatment, all participants were asked to give their overall feedback on the intervention, to which they could provide open-ended responses. demonstrates how various data sources contributed to the different components of implementation fidelity.
|Quantitative data sources|
|Demographic and clinical data||✓a||✓|
|Treatment credibility and satisfaction||✓|
|Qualitative data sources|
|Therapist and clinician interviews||✓||✓||✓||✓|
|End-of-treatment feedback questionnaire||✓|
aAssociated data with implementation component.
Data were tested for normality using the Kolmogorov–Smirnov test. A principal component analysis was run to determine a composite measure of the level of engagement. Correlations between variables were examined using bivariate Spearman correlations, and a t test was performed to explore any significant differences between groups using chi-square tests to explore the differences between categorical variables. Multiple linear regression was used to identify predictors of engagement with the independent variables. All statistical analyses used a significance level of P<.05 and were conducted using SPSS Statistics 27 (IBM Corporation).
All interviews were recorded by using either videoconferencing software or a Dictaphone and were then transcribed verbatim. Transcripts were checked for accuracy against the recordings, with any corrections made as appropriate and anonymized for confidentiality purposes. As the process evaluation was a combination of exploration and description, the framework method  of analysis was used to identify, analyze, and report patterns within the transcribed interviews. Moreover, the steps outlined by Gale et al [ ] were systematically followed to create an overall framework matrix using categories of engagement and potential moderators. Consistency of analysis was ensured through the use of a codebook and frequent meetings between researchers. Researcher bias was minimized through regular cross-checking of data and outcomes by the members of the research team.
The NVivo 12 software package (QSR International) was used to analyze the interview data. In addition, the end-of-treatment feedback questionnaire was exported to a Microsoft Excel spreadsheet, and quantitative content analysis  was performed. Overall, the findings from the qualitative analysis were linked to relevant quantitative engagement outcomes and contextual factors to assess which potential moderators may have influenced implementation fidelity and in what way, in an approach termed triangulation [ ].
Availability of Data and Materials
Some of the data generated or analyzed during this study are included in this paper and its supplementary information files or are available from the corresponding author upon reasonable request. The full data sets generated or analyzed during this study are also available from the corresponding author upon reasonable request.
Ethical approval for the study was obtained from the North West Greater Manchester Central Research Ethics Committee (REC: 18/NW/0079). We sought written parental consent and written informed assent or consent to participate in the study from CYP. This covered the process evaluation measures. All participants provided verbal consent to be audio-recorded for all interviews.
Overview of Qualitative Sample
Semistructured interviews were conducted with children (n=20), parents (n=20), therapists (n=4), and clinicians (n=6). The average age of the child interviewees was 12 years (SD 2.1); range 9-16 years), of which 80% (16/20) were male and 20% (4/20) were female. Most participants were White (18/20, 90%). The mean TTSS was 28.8 out of 50 (SD 7.2), with a range of 13-45 for the child interviewees. All 20 interviews with the parents were with the CYP’s mothers, with all 20 having completed at least some further education. One of the therapist interviewees was a therapist’s supervisor, and half of the clinicians (3/6, 50%) were consultant psychiatrists.
Participants were eligible for the study if they were aged between 9 and 17 years and competent to provide written, informed consent (parental consent for a child aged <16 years), had a suspected or confirmed tic disorder (as confirmed by scores on the YGTSS), and had broadband internet access and regular use of a computer, with mobile phone text messaging facilities. Patients were excluded from the study if they had received any form of structured behavioral intervention for tics within the preceding 12 months, had a change of medication for tics within the previous 2 months, had any diagnoses of alcohol or substance dependence, psychosis, suicidality, anorexia nervosa, or moderate or severe intellectual disability, were an immediate risk to self or others, and/or parents or children were not able to speak, read, or write English.
A total of 445 families expressed an interest in participating in the study either through self-referral via the Tourettes Action charity website (n=251) or clinic referral (n=194); however, 47 were subsequently uncontactable, and 90 were ineligible to participate. Of the 308 potentially eligible CYP, 84 (27.3%) families declined to participate, and 50% (112/224) of CYP (90 males and 22 females) with a mean age of 12.2 (SD 2) years were randomized to the intervention arm of the ORBIT trial and included in the process evaluation. The sample was predominantly White (96/112, 85.7%) and well-educated, with more than half (60/112, 53.5%) of the participants’ mothers having completed university or higher education.
The median IMD rank was 19,318 and ranged from 147 to 32,668 (out of 32,844). Of the 112 participants, 8 (7.1%) were in the most deprived quintile 1, 31 (27.7%) in quintile 2, 18 (16.1%) in quintile 3, 26 (23.3%) in quintile 4, and 29 (25.6%) in the least deprived quintile 5. Although the reach of the intervention was not limited geographically, for research purposes, participants had to attend a baseline screening assessment at either the Nottingham study site (57/112, 50.9%) or the London study site (55/112, 49.1%) depending on personal preference and/or location of residence. All participants were based in England, with 56.3% (63/112) of participants living in towns, 26.7% (30/112) living in cities, and 16.9% (19/112) living in villages.
In terms of clinical characteristics, the intervention reached a moderately severe symptomatic sample with a mean TTSS of 28.4 out of 50 (SD 7.7) ranging from 12 to 50. Most participants (98/112, 87.5%) were not on any medication for their tics, and less than half of the overall intervention sample had no diagnosed or suspected comorbidities (51/112, 45.5%). Among those who had a comorbid diagnosis, the most common was anxiety disorder (34/112, 30.4%), followed by attention-deficit/hyperactivity disorder (26/112, 23.2%). An assessment of depressive symptoms by the MFQ showed a mean score of 16.3 out of 66 (SD 11.3), with 12.5% (14/112) of participants scoring above the cut-off (≥29), suggesting clinically significant depression  ( ).
It was not possible to interview people who had not taken part in the study, so the qualitative data threw little light on reach. However, a clinician identified that some families were worried about the level of commitment involved and associated travel to one of the study sites (quote 1) under the theme clinician perceptions of and contribution to recruitment (seefor a full list of framework categories and themes and for a full list of quotes). Another clinician highlighted the lack of access to children with intellectual disabilities (quote 2). Finally, one of the clinicians struggled to gain her colleagues’ interest in the intervention despite numerous attempts (quote 3).
|Gender, n (%)|
|Study site, n (%)|
|Ethnicity, n (%)|
|Mixed race||3 (2.7)|
|Supporter, n (%)|
|Highest level of education (mother), n (%)|
|Did not complete compulsory education||3 (2.7)|
|Completed compulsory secondary education||16 (14.3)|
|Completed further education||33 (29.5)|
|Completed university or higher education||43 (38.4)|
|Completed postgraduate taught degree||11 (9.7)|
|Completed doctorate or medical degree||6 (5.4)|
|Highest level of education (father), n (%)|
|Did not complete compulsory education||2 (1.8)|
|Completed compulsory secondary education||29 (25.9)|
|Completed further education||35 (31.2)|
|Completed university or higher education||29 (25.9)|
|Completed postgraduate taught degree||10 (8.9)|
|Completed doctorate or medical degree||7 (6.3)|
|Method of referral, n (%)|
|Age (years), mean (SD)||12.2 (2)|
|IMDa rank, median (range)||19318 (147-32,668)|
|No tic medication, n (%)||98 (87.5)|
|On tic medication, n (%)||14 (12.5)|
|Comorbidities, n (%)||61 (54.5)|
|No comorbidities, n (%)||51 (45.5)|
|TTSSb baseline score, mean (SD)||28.4 (7.7)|
|MFQc, mean (SD)||16.3 (11.3)|
aIMD: index of multiple deprivation.
bTTSS: Total Tic Severity Score.
cMFQ: Mood and Feelings Questionnaire.
Child participants completed an average of 7.5 (SD 2.7;) chapters, and their parents completed an average of 7.6 (SD 2.8; ) out of 10 chapters of the intervention, indicating high engagement. Only 11.6% (13/112) of child participants and 15.2% (17/112) of parents failed to meet the criteria for treatment completion (ie, the minimum of the first four chapters completed as per protocol) with a total of 88.4% (99/112) of child participants and 84.8% (95/112) of parents completing their treatment, meaning that adherence to the intervention was high. Indeed, 41% (46/112) of CYP and 46.4% (52/112) of parents completed all 10 intervention chapters, and only 1 child participant failed to complete any chapters. Participants were given 10 weeks of supported therapeutic input to complete their treatment chapters. In some circumstances, such as holidays or particularly busy periods, 1 or 2 weeks were added to supplement this time. Although most families (73/112, 65.2%) completed their therapy within 10 weeks, 34.8% (39/112) required extra time to complete treatment. Child participants logged onto the web-based treatment platform an average of 19.8 (SD 10.9) times throughout the 10-12 weeks, with an average of 4.2 (SD 2.6) days between log-ins. In terms of total interactions with their assigned therapist, child participants required their therapist’s assistance on the web for an average of 59 minutes and 14 seconds (SD 29 min and 08 s) throughout treatment, resulting in approximately 6 minutes per child each week. In contrast, parents interacted on the web with their assigned therapist for an average of 1 hour, 23 minutes, and 55 seconds (SD 42 min and 45 s), resulting in approximately 8 minutes per parent each week. Of the 112 CYP, only 2 (1.8%) were contacted by telephone by their assigned therapist, whereas of the 112 parents, 49 (43.7%) were contacted by telephone by their assigned therapist.
Interview data relating to participants’ perceptions of the ORBIT organization covered the implementation component of the dose. Although most participants felt that the intervention was just the right length, some CYP wished to have a longer time to access their therapist (quote 4). A child felt that the intervention could have been condensed to make it shorter (quote 5). On the whole, parents agreed with their child that the dose received was just right, with a parent claiming that if it were longer, it would have negatively affected engagement levels (quote 6).
|Median (range)||Mean (SD)|
|Chapters completed||8 (0-10)||7.5 (2.7)|
|Total therapist time (hh:mm:ss)||00:53:57 (00:07:27-03:11:08)||00:59:14 (00:29:08)|
|Telephone time with therapist (hh:mm:ss)||00:00:00 (00:00:00-00:18:44)||00:00:10 (00:01:46)|
|Number of log-ins||19 (3-57)||19.8 (10.9)|
|Number of days between log-ins||3 (1-16)||4.2 (2.6)|
|Number of pages visited per log-in||15 (7-38)||16.9 (5.8)|
|Total number of characters submitted||2507 (238-8749)||2784 (1608)|
|Median (range)||Mean (SD)|
|Chapters completed||9 (1-10)||7.6 (2.8)|
|Total therapist time (hh:mm:ss)||01:15:33 (00:22:01-04:48:19)||01:23:55 (00:42:45)|
|Telephone time with therapist (hh:mm:ss)||00:00:00 (00:00:00-00:49:00)||00:04:06 (00:07:41)|
|Number of log-ins||18 (3-50)||20.4 (11.4)|
|Number of days between log-ins||4 (0-19)||4.2 (2.7)|
|Number of pages visited per log-in||17 (9-36)||17.4 (5.2)|
|Total number of characters submitted||6533 (346-29,631)||7286 (5093)|
At the 3-week point of postrandomization, participants were asked to rate treatment credibility. Treatment credibility was rated highly by the child participants, with a mean score of 6.4 out of 8 (SD 1.5). Furthermore, at the primary endpoint, participants were asked to rate their overall satisfaction with the intervention. Child participants were highly satisfied with the intervention, with a mean score of 24.8 out of 32 (SD 5.2). At the end of the treatment, participants were asked to give their feedback on the intervention within the web-based platform, and they were able to give open-ended responses. Only 59.8% (67/112) of child participants provided feedback. From the conducted quantitative content analysis, four categories emerged relating to implementation fidelity, namely limitations of ORBIT (51/67, 76%), which captured how participants felt that overall ORBIT was helpful however was limited by certain factors; ORBIT as a suitable treatment (49/67, 73%), which suggested that participants felt that the web-based delivery of treatment for tic disorders was suitable; problems with using ORBIT (20/67, 30%), which captured those participants who stated that they felt ORBIT was not helpful to them or was associated with negative factors; and feeling supported (19/67, 28%; seefor a full list of content analysis categories and codes), where participants mentioned that they felt supported in a way they had never been before (eg, by their therapist). The main code relating to limitations of ORBIT centered on improvement required (n=33). This code captured anything related to the intervention being unhelpful or inappropriate. Examples include repetitiveness of treatment, the treatment being too short or too long, unhelpful aspects, and suggested improvements. Two child participants reported technical issues with the ORBIT platform, which was related to intermittent problems with connectivity. Despite this, many participants felt the intervention was acceptable as a treatment, with the largest number of participants being coded as a positive experience of ORBIT (n=42), which was part of the category ORBIT as a suitable treatment and related to being pleased by having taken part and finding it enjoyable while recommending the treatment to other CYP with tic disorders.
Although satisfaction was rated highly, some participants felt that the role of the therapist was somewhat misleading. This was captured by the theme expectations of role of the therapist. Some felt that a therapist was not needed for the delivery of the intervention (quote 7). Some parents agreed with the sentiment that they could have completed the therapy without the assistance of a therapist (quote 8).
The term therapist itself was felt to be somewhat misjudged as a label, and they viewed the therapist more as a motivator (quote 9). The therapists themselves concurred with this, and perhaps they should not have been called a therapist within the ORBIT study (quote 10).
At the end of the interviews, participants were asked if they had any recommendations to improve the intervention, and the overriding majority felt that a mobile application was needed in future iterations of the intervention (quote 11). Some of the older CYP felt that the content and presentation of the intervention were childlike and aimed more toward younger participants. Therefore, they felt there could be two separate versions: one for teenagers and one for young children (quote 12).
Regarding adaptations, the intervention did not appear to evolve in any way from the original plans. Instead, there appeared to be consistency in the way the intervention was delivered and received. Interviews with therapists confirmed that consistency was maintained during delivery. For example, they created a list of standardized responses to common queries (quote 13). Parts of the intervention were designed to be adapted by the user and tailored to their needs and preferences, such as the tic stopwatch and tic ladder (hierarchy of exposure exercises), which was confirmed in the interviews (quote 14). Another participant adapted the intervention to make it easier to complete by altering some activities to make them more user friendly (quote 15).
Contextual Factors Influencing Intervention Implementation
To establish a measure of intervention implementation that captured both the breadth and depth of participants’ usage, a principal component analysis with varimax (orthogonal) rotation was conducted on the seven items related to the dose of intervention received. The analysis suggested a two-factor model. The strongest factor accounted for 47% of the variance (Eigenvalue 3.3;) and seemed to capture the strength of engagement with the intervention. Factor scores ranged from −2.65 to 2.26 with a mean of 0.001 (SD 0.99), and these scores were used as the engagement measure.
A two-tailed t test found that participants enrolled at the London site (mean 0.25, SD 0.90) scored significantly higher on engagement than those enrolled at the Nottingham site (mean −0.22, SD 1.03; t109=−2.58; P=.01). Moreover, those who were self-referred (mean 0.16, SD 0.94) scored higher on engagement than those referred by clinics (mean −0.24, SD 1.04; t109=−2.06; P=.04). Spearman rho (ρ) correlations were used to determine the association between engagement and various contextual factors. CYP’s engagement factor scores were strongly correlated with parents’ chapter completion (n=111; ρ=0.73; P<.001) and moderately correlated with therapist time for parents (n=111; ρ=0.46; P<.001). There were no significant relationships between the CYP’s engagement factor score and age, parental education, IMD, TTSS at baseline, or MFQ baseline score. There were also no statistically significant relationships among the child’s gender, comorbidities, or use of tic medication and CYP’s engagement.
Multiple linear regression was conducted, with CYP’s engagement factor score as the dependent variable, and site, child’s age, child’s gender, IMD, TTSS, referral method, parental education, therapist time for parents, and parents’ chapter completion as the independent variables. The results of the simultaneous regression indicated that, collectively, the independent variables had a significant amount of variance in the CYP’s engagement factor score (F10,100=20.84; P<.001; R2=0.64). There was no evidence of multicollinearity, with all tolerances above 50% and all variance inflation factors below 2. Only parents’ chapter completion (β=.69; t110=10.18; P<.001) and therapist time for parent (β=.19; t110=2.95; P=.004) were significant independent predictors in the model.
The theme of parental persuasiveness was generated in the framework category participant contextual factors. Many of the parents interviewed outlined that they were often the main motivating force behind their child’s level of engagement by reminding their child to practice the learned techniques (quote 16). Some parents found motivating their child to engage very challenging (quote 17), and this was even more challenging for those with children who have comorbidities (quote 18).
Some parents found it difficult to support their children because of hectic schedules, which was captured by the theme of busy lives (quote 19). Although under the theme of high motivation levels, we found highly engaged CYP without their parents’ persuasion (quote 20).
|Factor 1: engagement||Factor 2: sporadic use|
|Number of log-ins||0.90||—a|
|Total therapist time per child||0.76||—|
|Total number of characters submitted||0.74||—|
|Number of days between log-ins||−0.63||0.54|
|Number of pages visited per log-in||−0.41||0.80|
|Telephone time with the therapist||−0.44||−0.46|
|Percentage of variance||47||21|
aItem not loaded onto factor.
This process evaluation used a mixed methods approach to investigate the extent to which the ORBIT intervention was implemented as planned within the RCT and explore participants’ experiences with the intervention and the contextual factors influencing children’s engagement. Doing so made it possible to identify reasons for variation in uptake, usage, and engagement, reflect on how implementation may ultimately give us greater confidence in the outcomes, and outline lessons for potential future implementation within routine care. The uptake of the intervention was high, with 88.4% (99/112) of participants receiving the predefined minimum effective dose of the first four chapters completed. The median uptake was eight chapters, and only 1 child failed to access any chapters. Fidelity of delivery was also excellent, with participants reporting high levels of satisfaction and acceptability.
The intended sample of CYP with a diagnosed tic disorder was reached, with 7.1% (8/112) of families residing in the most deprived areas (IMD quintile 1) and more than a quarter (29/112, 25.6%) of the families residing in the least deprived areas (IMD quintile 5). As more than half (60/112, 53.5%) of the CYP’s mothers had completed university education, against a UK average of 42% , it seems that more advantaged families may have been overrepresented. Perhaps the requirement to have broadband internet access and regular use of a computer with mobile phone text messaging facilities to participate in the study may have differentially impacted participants in the most deprived IMD quintile. This is a concern, as one of the aims of the ORBIT was to increase access to evidence-based therapeutic interventions for CYP with tic disorders. In particular, access to services is generally limited to those from lower economic backgrounds [ ]. However, the initial baseline visit with associated travel may have been a disincentive to more disadvantaged families—a limitation that would not be relevant if ORBIT was delivered entirely remotely in routine care rather than as part of an RCT. Moreover, there is no evidence that socioeconomic factors influence CYP’s engagement with ORBIT. Furthermore, children’s age, tics severity, well-being, and comorbidities did not appear to influence the child’s level of engagement with the intervention, providing further evidence that it would have a wide reach within routine clinical care. However, because of the various factors related to this RCT as opposed to routine care, caution should be taken when interpreting the results of this study concerning reach.
The London study site, self-referral, and higher parental engagement were all associated with higher levels of engagement. The London site is a world-renowned center of excellence for pediatric care, which may have increased parents’ motivation for treatment. However, the only independent predictor of child engagement in the multivariate analysis was the level of parental engagement with intervention, as measured by their chapter completion and time with the therapist. This is consistent with previous literature [- ], which found that parental involvement was particularly key for younger CYP to assist with their engagement with therapeutic interventions, which in turn leads to better outcomes [ - ]. It has been shown in the literature that parental engagement may impact a provider’s ability to implement parent- and family-focused evidence-based treatment with fidelity [ ]. Therefore, it is crucial to understand the role of parental support in the implementation of DHIs for children, as without attention to the key processes of child and family engagement, efforts to improve the effectiveness and efficiency of the treatment are less likely to succeed. Furthermore, it is crucial to assess whether parental support also predicts intervention efficacy and the mechanisms through which its impact is achieved.
An interesting finding was the use and interactions with the therapists in this study. Therapists interacted on the web with their assigned child participants an average of 6 minutes per child each week, which is lower than the 24 minutes average time per week participants interacted with their therapist in the Swedish pilot trial, on which ORBIT is based . However, in the United Kingdom, study therapists were encouraged to use preprepared scripts to respond to the participants. Their responsibilities involved reinforcing the ORBIT treatment material with the aim of spending around 6 minutes a week in response to each child. Detailed analysis of the content of therapists’ interactions is outside the remit of this study, but it is apparent from qualitative interviews that many participants felt that the term therapist was somewhat misleading. Some participants felt that therapist had connotations of a clinically-trained individual delivering an intervention. This may have limited their reliance on therapists. Therefore, in any implementation of this intervention within routine health care, it would be sensible to alter the title to coach, guide, or mentor as this better reflects the therapist’s role.
Strengths and Limitations
To the best of our knowledge, this study is one of the first to conduct an in-depth mixed methods process evaluation of a complex intervention aimed at CYP with Tourette syndrome and chronic tic disorders. A number of important findings emerged from the process data, which helped us characterize the intervention’s implementation within an RCT and provided lessons for potential future implementation within routine care. However, these lessons can only be fully realized once the main RCT outcome data have been analyzed. Furthermore, a principal component analysis of participants’ usage data provided an objective, reliable, and comprehensive measure of engagement to explore the role of contextual factors.
However, this study has some limitations. First, there was the issue of potential recruitment bias. It may have been that the more motivated families self-referred to the trial and that recruitment from clinics was skewed toward punctual, frequent attenders, in contrast to patients with multiple missed appointments. This may have limited the power of this process evaluation to detect socioeconomic biases in engagement. Second, although comprehensive, the information on uptake cannot fully capture the quality and quantity of engagement with the ORBIT. For example, indices such as chapter completion, number of pages visited, and number of log-ins may not fully capture factors such as level of attention to practice exercises. Finally, and perhaps most crucially, a major limitation was that it was not possible to interview those who had not taken part in the RCT or reach those who had withdrawn early from the study. Their perspective is vital to fully understand the factors influencing engagement with DHIs.
We conclude that the intervention had high fidelity of delivery and was evaluated positively by CYP, although some participants suggested some minor improvements, and the nature of the RCT may have constrained reach. Parental engagement was a strong, independent predictor of intervention implementation, which has important implications for designing and implementing digital therapeutic interventions in child and adolescent mental health services.
The authors thank Tourettes Action for their ongoing support with the ORBIT trial and particularly acknowledge Dr Seonaid Anderson for her help and advice. The authors also thank Patient and Public Involvement members for their help in revising the interview schedules, including James Bungay, Claire Bungay, Sandra Wang, and Marco Wang. Finally, the authors thank all participants who consented to be interviewed. All research at the Great Ormond Street Hospital National Health Service (NHS) Foundation Trust and University College London Great Ormond Street Institute of Child Health is made possible by the National Institute for Health Research (NIHR) Great Ormond Street Hospital Biomedical Research Center. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health. This process evaluation was funded by the NIHR MindTech MedTech Co-operative and NIHR Nottingham Biomedical Research Center Mental Health and Technology Theme. The ORBIT trial is funded by the NIHR Health Technology Assessment (ref 16/19/02). CG, EBD, and CH acknowledge the financial support of the NIHR Nottingham Biomedical Research Center and NIHR MindTech MedTech Co-operative. This research was supported by the NIHR Biomedical Research Center at the Great Ormond Street Hospital for Children’s NHS Foundation Trust and University College London.
CG, KK, CLH, and EBD contributed to the mixed methods process evaluation conception and developed the interview guides. KK conducted data collection and initial analysis of both quantitative and qualitative data. KK wrote the initial draft and the subsequent revisions. CG, CLH, and EBD provided a critical review and editing of the initial and subsequent drafts of the manuscript. All authors critically revised the manuscript and read and approved the final manuscript. CH is the principal investigator in the ORBIT trial, designed the trial, and wrote the original grant application.
Conflicts of Interest
Analytic framework.DOCX File , 19 KB
Qualitative quotes.DOCX File , 17 KB
Content analysis tables.DOCX File , 18 KB
- American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th Edition: DSM-5. Washington, DC: American Psychiatric Association Publishing; 2013:1-991.
- Lichter D, Finnegan S. Influence of gender on Tourette syndrome beyond adolescence. Eur Psychiatry 2015 Feb;30(2):334-340. [CrossRef] [Medline]
- Freeman RD, Fast DK, Burd L, Kerbeshian J, Robertson MM, Sandor P. An international perspective on Tourette syndrome: selected findings from 3,500 individuals in 22 countries. Dev Med Child Neurol 2000 Jul;42(7):436-447 [FREE Full text] [CrossRef] [Medline]
- Leckman JF, Zhang H, Vitale A, Lahnin F, Lynch K, Bondi C, et al. Course of tic severity in Tourette syndrome: the first two decades. Pediatrics 1998 Jul;102(1 Pt 1):14-19. [CrossRef] [Medline]
- Shprecher D, Kurlan R. The management of tics. Mov Disord 2009 Jan 15;24(1):15-24 [FREE Full text] [CrossRef] [Medline]
- Stroup TS, Gray N. Management of common adverse effects of antipsychotic medications. World Psychiatry 2018 Oct 07;17(3):341-356 [FREE Full text] [CrossRef] [Medline]
- McGuire JF, Piacentini J, Brennan EA, Lewin AB, Murphy TK, Small BJ, et al. A meta-analysis of behavior therapy for Tourette syndrome. J Psychiatr Res 2014 Mar;50:106-112. [CrossRef] [Medline]
- Whittington C, Pennant M, Kendall T, Glazebrook C, Trayner P, Groom M, et al. Practitioner review: treatments for Tourette syndrome in children and young people - a systematic review. J Child Psychol Psychiatry 2016 Sep;57(9):988-1004 [FREE Full text] [CrossRef] [Medline]
- Hollis C, Pennant M, Cuenca J, Glazebrook C, Kendall T, Whittington C, et al. Clinical effectiveness and patient perspectives of different treatment strategies for tics in children and adolescents with Tourette syndrome: a systematic review and qualitative analysis. Health Technol Assess 2016 Jan;20(4):1-450, vii-viii [FREE Full text] [CrossRef] [Medline]
- Novotny M, Valis M, Klimova B. Tourette syndrome: a mini-review. Front Neurol 2018;9:139 [FREE Full text] [CrossRef] [Medline]
- Hollis C, Falconer C, Martin J, Whittington C, Stockton S, Glazebrook C, et al. Annual Research Review: digital health interventions for children and young people with mental health problems - a systematic and meta-review. J Child Psychol Psychiatry 2017 Apr;58(4):474-503 [FREE Full text] [CrossRef] [Medline]
- Ricketts EJ, Goetz AR, Capriotti MR, Bauer CC, Brei NG, Himle MB, et al. A randomized waitlist-controlled pilot trial of voice over internet protocol-delivered behavior therapy for youth with chronic tic disorders. J Telemed Telecare 2016 Apr;22(3):153-162 [FREE Full text] [CrossRef] [Medline]
- Himle MB, Freitag M, Walther M, Franklin SA, Ely L, Woods DW. A randomized pilot trial comparing videoconference versus face-to-face delivery of behavior therapy for childhood tic disorders. Behav Res Ther 2012 Sep;50(9):565-570. [CrossRef] [Medline]
- Andrén P, Aspvall K, de la Cruz L, Wiktor P, Romano S, Andersson E, et al. Therapist-guided and parent-guided internet-delivered behaviour therapy for paediatric Tourette's disorder: a pilot randomised controlled trial with long-term follow-up. BMJ Open 2019 Feb 15;9(2):e024685 [FREE Full text] [CrossRef] [Medline]
- Hall C, Davies E, Andrén P, Murphy T, Bennett S, Brown B, ORBIT Trial team.. Investigating a therapist-guided, parent-assisted remote digital behavioural intervention for tics in children and adolescents-'Online Remote Behavioural Intervention for Tics' (ORBIT) trial: protocol of an internal pilot study and single-blind randomised controlled trial. BMJ Open 2019 Jan 03;9(1):e027583 [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [CrossRef] [Medline]
- Glasgow R, Lichtenstein E, Marcus A. Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health 2003 Aug;93(8):1261-1267 [FREE Full text] [CrossRef] [Medline]
- Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al. Evaluating digital health interventions: key questions and approaches. Am J Prev Med 2016 Nov;51(5):843-851. [CrossRef] [Medline]
- Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. Br Med J 2015 Mar 19;350(mar19 6):h1258 [FREE Full text] [CrossRef] [Medline]
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J 2008 Sep 29;337:a1655 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. The law of attrition. J Med Internet Res 2005 Mar 31;7(1):e11 [FREE Full text] [CrossRef] [Medline]
- Hermes E, Lyon A, Schueller S, Glass J. Measuring the implementation of behavioral intervention technologies: recharacterization of established outcomes. J Med Internet Res 2019 Jan 25;21(1):e11752 [FREE Full text] [CrossRef] [Medline]
- Hawe P, Shiell A, Riley T. Complex interventions: how "out of control" can a randomised controlled trial be? Br Med J 2004 Jun 26;328(7455):1561-1563 [FREE Full text] [CrossRef] [Medline]
- Bumbarger B, Perkins D. After randomised trials: issues related to dissemination of evidence‐based interventions. J Child Serv 2008 Apr 12;3(2):55-64 [FREE Full text] [CrossRef]
- Khan K, Hollis C, Hall C, Davies E, Mataix-Cols D, Andrén P, et al. Protocol for the process evaluation of the Online Remote Behavioural Intervention for Tics (ORBIT) randomized controlled trial for children and young people. Trials 2020 Jan 02;21(1):6 [FREE Full text] [CrossRef] [Medline]
- English indices of deprivation 2019: technical report. Ministry of Housing, Communities & Local Government, Government of UK. 2019. URL: https://www.gov.uk/government/publications/english-indices-of-deprivation-2019-technical-report [accessed 2021-06-07]
- Leckman JF, Riddle MA, Hardin MT, Ort SI, Swartz KL, Stevenson J, et al. The Yale Global Tic Severity Scale: initial testing of a clinician-rated scale of tic severity. J Am Acad Child Adolesc Psychiatry 1989 Jul;28(4):566-573. [CrossRef] [Medline]
- Costello EJ, Angold A. Scales to assess child and adolescent depression: checklists, screens, and nets. J Am Acad Child Adolesc Psychiatry 1988 Nov;27(6):726-737. [CrossRef] [Medline]
- Daviss WB, Birmaher B, Melhem N, Axelson D, Michaels S, Brent D. Criterion validity of the Mood and Feelings Questionnaire for depressive episodes in clinic and non-clinic subjects. J Child Psychol Psychiatry 2006 Sep;47(9):927-934 [FREE Full text] [CrossRef] [Medline]
- Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health 2015 Sep 6;42(5):533-544 [FREE Full text] [CrossRef] [Medline]
- Fusch P, Ness L. Are we there yet? Data saturation in qualitative research. Qual Rep 2015 Sep 8:A [FREE Full text] [CrossRef]
- Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Bryman A, Burgess B, editors. Analyzing Qualitative Data. London: Routledge; 1994:173-194.
- Gale N, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013 Sep 18;13:117 [FREE Full text] [CrossRef] [Medline]
- Hsieh H, Shannon S. Three approaches to qualitative content analysis. Qual Health Res 2005 Nov;15(9):1277-1288 [FREE Full text] [CrossRef] [Medline]
- O'Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. Br Med J 2010 Sep 17;341:c4587. [CrossRef] [Medline]
- Graduates in the UK labour market. Office for National Statistics. 2017. URL: https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/articles/graduatesintheuklabourmarket/2017#steady-increase-in-the-number-of-graduates-in-the-uk-over-the-past-decade [accessed 2021-06-07]
- Packness A, Waldorff F, Christensen RD, Hastrup L, Simonsen E, Vestergaard M, et al. Impact of socioeconomic position and distance on mental health care utilization: a nationwide Danish follow-up study. Soc Psychiatry Psychiatr Epidemiol 2017 Nov;52(11):1405-1413 [FREE Full text] [CrossRef] [Medline]
- Cobham VE. Do anxiety-disordered children need to come into the clinic for efficacious treatment? J Consult Clin Psychol 2012 Jun;80(3):465-476. [CrossRef] [Medline]
- Lyneham HJ, Rapee RM. Evaluation of therapist-supported parent-implemented CBT for anxiety disorders in rural children. Behav Res Ther 2006 Sep;44(9):1287-1300. [CrossRef] [Medline]
- Pennant ME, Loucas CE, Whittington C, Creswell C, Fonagy P, Fuggle P, et al. Computerised therapies for anxiety and depression in children and young people: a systematic review and meta-analysis. Behav Res Ther 2015 Apr;67:1-18. [CrossRef] [Medline]
- Thirlwall K, Cooper P, Karalus J, Voysey M, Willetts L, Creswell C. Treatment of child anxiety disorders via guided parent-delivered cognitive-behavioural therapy: randomised controlled trial. Br J Psychiatry 2013 Dec;203(6):436-444 [FREE Full text] [CrossRef] [Medline]
- Haine-Schlagel R, Walsh N. A review of parent participation engagement in child and family mental health treatment. Clin Child Fam Psychol Rev 2015 Jun;18(2):133-150 [FREE Full text] [CrossRef] [Medline]
- Nock M, Ferriter C. Parent management of attendance and adherence in child and adolescent therapy: a conceptual and empirical review. Clin Child Fam Psychol Rev 2005 Jun;8(2):149-166 [FREE Full text] [CrossRef] [Medline]
- Vernon T, Koegel R, Dauterman H, Stolen K. An early social engagement intervention for young children with autism and their parents. J Autism Dev Disord 2012 Dec;42(12):2702-2717 [FREE Full text] [CrossRef] [Medline]
|CYP: children and young people|
|DHI: digital health intervention|
|IMD: index of multiple deprivation|
|MFQ: Mood and Feelings Questionnaire|
|MRC: Medical Research Council|
|NHS: National Health Service|
|NIHR: National Institute for Health Research|
|ORBIT: Online Remote Behavioral Intervention for Tics|
|RCT: randomized controlled trial|
|TTSS: Total Tic Severity Score|
|YGTSS: Yale Global Tic Severity Scale|
Edited by R Kukafka; submitted 04.11.20; peer-reviewed by J van de Griendt, H McCall; comments to author 23.12.20; revised version received 15.01.21; accepted 16.03.21; published 21.06.21Copyright
©Kareem Khan, Chris Hollis, Charlotte L Hall, Elizabeth Murray, E Bethan Davies, Per Andrén, David Mataix-Cols, Tara Murphy, Cris Glazebrook. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 21.06.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.