There is mounting evidence from randomized controlled trials (RCTs) that parents can be trained to tackle and reduce children’s disruptive behavior and improve their parenting skills [1-3]. These findings are of upmost importance to public health professionals because children who exhibit disruptive behavior face increased risks of adult psychiatric disorders, substance use, crime, suicide, and other adversities [4,5]. Sufficiently strong evidence has been published on the efficacy of parent training to suggest that psychosocial services for children should include evidence-based parent training programs [6,7]. The need for services to tackle childhood disruptive behavior is enormous, but only a minority of families receive them [8]. There are challenges to implementing traditional face-to-face group-based parent training programs in real-world settings. One issue is the large number of barriers such as high cost, poor access, inconvenience, and low fidelity [3,9]. Another is keeping the content of the intervention consistent with the original evidence-based treatment [2].

Digitally assisted interventions are becoming more common, as they can overcome the barriers associated with conventional programs [3,9]. They are also likely to become increasingly popular, as child mental health services struggle to deal with the considerable increase in demand for their services as a result of the COVID-19 pandemic. This unprecedented global health emergency is expected to have major ongoing effects on child mental health owing to factors such as quarantine measures, social distancing, and school closures [10]. The pandemic started at a time when resources were already under pressure, and these are expected to be further affected by manpower shortages and a global recession that puts even greater pressure on health budgets. Digitally assisted interventions are cost-effective solutions that require fewer personnel and can reach geographically remote areas that would otherwise be outside of the reach of specialist services.

RCT studies have shown that remote and digitally assisted parent training programs have worked well in clinical settings [11,12]. We previously reported 12-month and 24-month follow-up studies of the first RCT on the Strongest Families Smart Website (SFSW). This RCT used a population-based sample and provided an internet-based parent training intervention with weekly telephone coaching [13-15]. The development of the SFSW intervention was based on the social learning and cognitive behavioral theories as well as positive parenting practices [16-18]. The target population was 4-year-old children who displayed high levels of disruptive behavior when they were screened during annual health checkups at child health clinics across Southwest Finland. The RCT showed that the children and parents who received the SFSW parent training program derived significant benefits from the initiative. The children displayed significant reductions in their disruptive behavior and other psychiatric symptom domains at their 24-month follow-up assessments. They also demonstrated the same improvements when they were compared with an education control group. The education control group received access to a static website that provided parents with information on how to tackle behavior problems and 1 phone call with a coach. Improved parenting skills were maintained in the intervention group at the 24-month follow-up assessment [14].

There has been growing interest in implementation research during the past 2 decades. Dissemination refers to how knowledge of new practices is actively and passively extended, and implementation refers to how new practices are incorporated into real-world environments. The term implementation gap is used to refer to the difference between our knowledge of what works and how it works [19,20]. Unfortunately, the strong effects that are observed in controlled RCT settings can weaken or become ambiguous when they are implemented in real-world settings [9]. Meta-analyses have shown that effective implementation has been associated with better outcomes, and the magnitude of the mean effect sizes was considerably higher when programs were carefully implemented and when fidelity was confirmed [21]. Successfully converting psychosocial interventions from experimental environments to real-world practice requires a solid framework and a structured implementation plan [22]. Research on evidence-based parent training programs after the RCT stage has often focused on examining the characteristics of an optimal implementation environment rather than maintaining the effectiveness of the intervention. We are not aware of any previous reports on the effectiveness of implementing digital interventions for disruptive behavior so that they can form part of the routine care that children below school age can receive.

This was the first study to report the effectiveness of the SFSW internet-based and telephone-assisted parent training program for preschool children when it was implemented in real-world settings. The intervention was put into practice after the population-based screening was used to identify children with disruptive behavior problems during routine visits to Finnish child health clinics at the age of 4 years. The primary aim was to report the changes in the children’s psychopathology and functioning level and any improvement in their families’ parenting skills. The children and their parents were followed up 6 months after the SFSW intervention was nationally implemented in Finnish primary care child health clinics. We compared the treatment characteristics and effectiveness between the families who received the SFSW intervention in these real-world settings from January 2015 to May 2017 and the families who received the intervention during the RCT from October 2011 to November 2013. Finally, we verified the findings by carrying out the following additional analyses. The first analysis excluded families who did not complete the parent training program. The second analysis excluded the Turku study site from the implementation study group because it was the only site that participated in both the RCT intervention and the implementation phases. In the third analysis, we compared the implementation and the RCT education control group. Our hypothesis was that the effectiveness of the SFSW intervention would be maintained if the protocol used in our previous RCT and the structured implementation plan were strictly adhered to.

This study was a longitudinal comparison of 2 parallel groups. The implementation group comprised 600 families who received the SFSW internet-based and telephone-assisted parent training program in the real-world setting between January 2015 and May 2017. The implementation phase covered 95 child health clinics in 12 administrative regions across Finland. The RCT intervention group comprised 232 families who had been recruited by 42 child health clinics in 7 administrative regions in Southwest Finland between October 2011 and November 2013. The administrative regions in both the RCT and implementation studies contained both urban and rural areas. Turku was the only region that participated in both studies.

There were both differences and similarities between the implementation and the RCT intervention studies. First, the implementation group received the intervention when it was integrated as a normal practice of the child health clinics, and therefore, all families who met the inclusion criteria were eligible to enter. In the implementation phase, both participants and the health care workers received information that the SFSW parent training intervention has been evaluated as an intervention with strong documented effects by the Finnish national evaluation and classification system for evidence-based interventions [23]. This evaluation was partly based on the results of our previous RCT study [13,14]. In contrast, in the RCT, the intervention was not integrated as a normal practice of the child health clinics. Only those families who were randomized to the intervention group received the intervention. Second, in the implementation phase, an implementation plan, including decision supporting and administration component, was followed. This was important because the implementation phase included increasing number of communities in the whole Finland while the RCT was conducted in a predetermined area of Southwest Finland. Third, the most important similarity was that the content of the SFSW intervention was maintained in the implementation group as identical as possible with that of the original RCT intervention. In both groups, the same psychopathology and parenting measures were collected at baseline and 6-month follow-up. Data on children’s daily activities were collected only for the implementation group immediately after the intervention and at the 6-month follow-up. The timeline of the RCT and implementation studies is shown in Figure S1 of Multimedia Appendix 1. The study protocol of the RCT has previously been published [24] and registered at ClinicalTrials.gov (NCT01750996).

This study focused on the 6-month follow-up assessments of children who displayed a high level of disruptive behavior when they were screened at 4 years of age during routine child health clinic visits. The screening procedure in the implementation study followed the same principles that were used in the RCT study. It was integrated into the standard 4-year-old child health checkups carried out by the child health clinics in the participating administrative regions [13]. All children living in Finland are invited to annual health assessments before they start school at 7 years of age, and attendance rates are just under 100% [25].

In the implementation group, the first 600 eligible parents who agreed to take part in the program received the SFSW parent training intervention. Initially, 8866 children were screened for highly disruptive behavior and 1099 (12.4%) met the screening criteria. The implementation group equated to 6.8% (600/8866) of the initial population-based sample and 54.6% (600/1099) of those who were eligible to take part. The reference group consisted of 232 families who were randomized to receive the intervention during the previous RCT study [13,14]. Information was obtained from 427 (71.2%) of the 600 families in the implementation group at the 6-month follow-up assessments compared to 184 (79.3%) of the 232 families in the RCT intervention group. Figure 1 shows the flowchart of the implementation and RCT intervention groups. The families were typically recruited within 1 month of the child’s fourth birthday. They received a study information pack and were asked to bring the completed health questionnaire to the clinic.

The screening measures and enrollment criteria were identical for the implementation and RCT studies [13]. Population-based screening for behavior problems was conducted for all children at the age of 4 years by using the conduct scale of the Strengths and Difficulties Questionnaire [26-28]. The parents were asked if their child had mild, moderate, or severe problems through a single question: “Overall, do you think that your child has difficulties in one or more of the following areas: emotions, behavior, or being able to get on with other people?” About 16.5% of the children (16.7% [1477/8866] and 16.3% [758/4656] in the implementation and RCT groups, respectively) who were screened, scored 5 or more out of 10 corresponding to the 80th percentile cutoff point and reported that the child had difficulties. This indicated a high level of behavior problems. The other inclusion criteria were that the parents perceived that child had at least minor difficulties in emotions, behavior, or social interactions. To participate in the study, the family had to live in an administrative region participating in the study, at least one parent had to speak native Finnish or Swedish, and they needed access to a telephone, computer, and internet connection. We excluded children who were unable to speak in full sentences, had hearing or vision impairments, or were receiving or had received behavior treatment. The exclusion criteria also included children who had been diagnosed with autism, Down syndrome, fetal alcohol syndrome, an intellectual disability, a severe mental disorder such as psychosis or depression, or who had a genetic diagnosis of mental retardation. We also excluded parents whose children did not live with them because they were subject to child protection services owing to child custody, abuse, or neglect issues. Details of the inclusion and exclusion criteria have previously been reported [24].

The participants in the implementation and RCT intervention study groups received the SFSW parent training program, which combines an interactive website with weekly telephone coaching [15,29]. One parent was identified for each child and they filled in the web-based questionnaire. However, they were also encouraged to get the child’s other parent involved in the program as much as possible. The program was guided by coaches who were professionals of health care and social services, that is, public health nurses, public nurses, or social workers from the child services. Of note, in the RCT study, there were 6 coaches, and during the implementation study, there were 10 coaches, 6 of whom had not participated in the RCT. The coaches had weekly phone calls with the parents, which were sometimes organized using texts or emails, and they monitored their progress on the website. The intervention consisted of 11 weekly themes that were explored during the interactive web-based program and the associated telephone coaching sessions. After the baseline survey, the coaches called the parents and they agreed to personalize goals tailored to individual behavior problems demonstrated by the child. The program aimed to reduce the problems identified by the parents by teaching them positive and practical parenting skills. During the first 7 weeks, the parent learned positive and practical problem-solving skills and were encouraged to develop an understanding of their child’s emotional development. The primary aim was to reorient the parent so that they noticed the child’s positive, not negative, behavior and reacted with a positive response. The second aim was to apply the skills in everyday situations, to plan daily activities in advance, and to use the methods they were taught to reinforce positive behavior. The final weekly themes focused on reinforcing their new skills and developing sustained positive parenting. The parents practiced the acquired skills with their child, independent of the coach’s support, and learned how to sustain the skills once the program had finished. The content and the conceptual framework of the weekly themes are depicted in Table 1. Each internet-based session comprised an introduction to the weekly theme, session content, video exercises, troubleshooting tips, and a review of what the parent had learnt. Instructional videos and audio clips illustrated the practical applications of the parents’ new skills. The coaches gave the parents feedback about their progress in applying the new skills and encouraged them throughout the program. They only proceeded to the next weekly theme when the parents had mastered the skill-related questions in the current one. This typically took 1 or 2 weeks. The children did not have access to the website or take part in the coaching calls. We are not aware of any potential adverse effects of the parent training in this study or in previous studies [30].

To ensure the integrity of the intervention and the accuracy of the data, several quality assurance measures were in effect during the implementation phase. These were similar to the quality assurance measures during the RCT study [13,14]. The implementation plan is summarized below and has been previously described in detail [15]. The implementation plan was driven by 3 core components [15,19]. First was recruitment, staff selection, and training. Once the coaches were recruited, they received intensive training on the SFSW program and were supervised and regularly monitored to make sure they adhered to the protocol. Together with supervision and staff performance evaluation, this provided systematic quality assurance [15]. The second core component was ongoing supervision and staff performance evaluation. The coaches took part in systematic weekly supervision meetings and group case conferences, where they reviewed and discussed the families they were coaching. Coaches with previous experience of the SFSW program acted as supervisors. After each telephone call, the coaches assessed their own performance on a scale of 4-10. The supervisor received a message from the digital platform about self-assessments that scored 6 or more and discussed the content of the call with the coach. To ensure the fidelity of the data, about 10% of the phone calls was audited by the coach supervisors with the parent’s permission and evaluated for competency. Additional training and monitoring of future calls were provided, if indicated. The coaches were required to report any adverse effects such as safety issues, abuse, or neglect to the supervisors, and the case was reported to the child protective services. Of note, 3 cases were reported during the implementation study and none during the RCT. The third core component was the decision supporting and administration. The development, delivery, and implementation process of the digital SFSW parent training intervention were centralized at the Research Center for Child Psychiatry at the University of Turku. The research group and the assisting staff of the Research Center introduced the SFSW intervention and the implementation process to the directors of child and family health services of the primary health care of each administrative region. A jointly funded research contract was signed by both parties. The research group maintained contact with the directors across the study region by organizing regular meetings and providing them with user-friendly monthly progress reports, which included the number of families who had been screened and enrolled. Training was offered to the team leaders of the child health clinics and public health nurses in order to integrate the intervention into primary health care. Moreover, local and national media were involved to increase public awareness of the SFSW intervention.

The analyses compared the 600 families in the real-world implementation group to the 232 families in the RCT intervention group. Categorical demographic variables, including the child, parent, and family characteristics, are presented as numbers and percentages. Continuous demographic variables, including the parents’ age and duration of child’s behavioral problems, are presented as means and standard deviations. We explored any differences at baseline between the 2 groups by using Pearson chi-square test or Fisher exact test for the categorical variables and the two-tailed Student t test for the continuous variables. The primary and secondary outcome variables were analyzed with a linear mixed-effect model for repeated measurements. The within factor was time, namely, baseline and 6-months follow-up, and the RCT intervention group and the implementation group provided the between factor. The covariates in the statistical models were CBCL externalizing scores at baseline, maternal education, duration of behavior problems, and the baseline measurement of each outcome. The statistical model used to analyze the CBCL externalizing score consisted of the group and time main effects, the group-by-time interaction effect, and the following covariates: the CBCL externalizing score at baseline, maternal education, and duration of behavior problems. Meanwhile, the statistical model used to analyze all the secondary outcome variables, namely, the CBCL total and other CBCL subscores, ICU, the Parenting Scale, and DASS-21 consisted of the group and time main effects and the group-by-time interaction effect. It also included the following covariates: the specific secondary variable to be analyzed at baseline, the CBCL externalizing score at baseline, maternal education, and the duration of behavior problems.

The sensitivity analyses comprised the families who had completed the parent training program as well as the treatment comparisons. Turku was excluded from analysis, as it was the only site that had taken part in both the implementation and RCT intervention studies. As the study subjects in the implementation group were recruited from January 2015 to May 2017 and in the RCT intervention group from October 2011 to November 2013, we also tested the effect of the recruitment year on the CBCL externalizing score at baseline. The model included the effects of recruitment year, maternal education, and duration of behavior problems. The effect of the recruitment year was insignificant (P=.17). An additional analysis also compared the implementation to the RCT education control group. The model included the CBCL externalizing score at baseline, maternal education, duration of behavior problems, paternal age, and the baseline measurements of each outcome as covariates. A P value <.05 was considered to be statistically significant. The statistical analyses were performed using SAS 9.4 (SAS Institute).

Ethical approval for the implementation study was received from the University of Turku (approval number: 18/2018). The parents provided written informed consent for both the implementation and the RCT studies.