Published on 11.04.19 in Vol 21, No 4 (2019): April
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/11446, first published Jun 29, 2018.
Web-Based Parent Training Intervention With Telephone Coaching for Disruptive Behavior in 4-Year-Old Children in Real-World Practice: Implementation Study
Background: Parent training is the most effective approach to the psychosocial treatment of disruptive behavioral problems in childhood. However, no studies exist on how well Web-based training programs work when they make the transition from the research setting to implementation in primary health care.
Objective: The study aimed to examine how the randomized controlled trial (RCT) and implementation study groups of the Strongest Families Smart Website (SFSW) intervention differed in child psychopathology, family demographics and treatment-related factors, such as therapeutic alliance and parents’ satisfaction rates. The intervention was conducted in the pediatric primary health care in Finland.
Methods: The study focused on 232 parents who had taken part in the SFSW intervention, which formed part of a 2-arm RCT study, and 882 families that would participate in the subsequent SFSW implementation study group. Both groups comprised parents whose children displayed high levels of parent-reported disruptive behavioral problems when they were screened in child health clinics at 4 years of age. Parents in both groups were provided with the SFSW intervention, which consisted of a Web-based training program with 11 weekly themes and associated telephone sessions.
Results: Demographic factors or duration of behavioral problems did not differ statistically or clinically between the RCT and implementation groups. Overall, 42.0% (362/862) of children in the implementation group and 35.4% (80/226) in the RCT intervention group had suffered from behavioral difficulties more than 1 year before the screening phase (χ12=3.2; P=.07). The mean duration of telephone coaching calls was very similar in the implementation and RCT intervention groups, that is, 38 and 37 min per call, respectively (t279.5=0.26; P=.79). The total time spent on the website of the program was 451 min in the implementation group and 431 min in the RCT intervention group (t318.8=1.38; P=.17). In the RCT intervention group, 52 of the 232 participants (22.4%) discontinued the program before the tenth week, whereas in the implementation group, 109 of the 882 participants (12.4%; odds ratio 2.05, 95% CI 1.4-3.0; P<.001) discontinued. Parents in both the implementation (77.1% to 98.5%, 498/742 to 731/742, respectively) and the RCT (64.8% to 98.2%, N=105/162- to 159/162, respectively) groups reported qualitatively similar and high level of posttreatment satisfaction rates in improved parenting skills, expectations, and stress relief. Parents in both groups reported a high level of satisfaction in skills and professionalism of the telephone coaches.
Conclusions: The implementation of population-based screening of Web-based parent training intervention with telephone coaching resulted in good feasibility, fidelity, accessibility, and similar satisfaction level post treatment when compared with intervention in RCT research setting. The discontinuation of treatment in the implementation group was exceptionally low.
J Med Internet Res 2019;21(4):e11446
Parent training is the most effective approach to the psychosocial treatment of disruptive behavioral problems in childhood, and there is mounting evidence from randomized controlled trials (RCTs) that such initiatives reduce problems and improve parenting skills [- ]. It has been proposed that parent training should be provided as primary promotion and prevention in primary care settings [ ]. However, only a small proportion of families with these problems benefit from evidence-based treatment programs [ ]. The barriers to receiving parent training include the lack of trained staff that can provide interventions; the stigma related to receiving mental health treatment; and the difficulties of accessing and engaging in treatment in terms of costs, time, and location [ , ].
Technology-based parent training programs can offer many benefits over traditional interventions, such as higher fidelity, greater accessibility, convenience, and reduced time and costs [- ]. Technology-based parent training is not a new innovation because, as early as 1988, Webster-Stratton et al [ ] tested videotapes as the primary delivery method for a parent training intervention. Today, technology and internet are integral parts of people’s lives. For example, in 2017, 88% of Finnish people used the internet, including almost every adult under the age of 55 years, and 77% had smartphones, which is one of the easiest ways to access the internet [ ]. Accordingly, in the field of parent training as primary care of children’s conduct problems, very recent research has shifted to focus on Web-based training programs. Studies have shown promising efficacy of Web-based interventions in improving child behavior [ , ]. Especially, interactive Web-based programs have been found to be more effective than noninteractive programs [ ]. Such Web-based interactive parent training programs could overcome many barriers associated with most traditional programs in the implementation phase, especially concerning the consistency with the original evidence-based intervention [ ].
We previously reported the 12-month follow-up study of the first RCT to provide an interactive Web-based parent training program with supplementary weekly phone coaching, the Strongest Families Smart Website (SFSW), using a population-based screening procedure . The 12-month follow-up study showed that the intervention resulted in significant reductions in the level of disruptive behavior problems among 4-year-old children and improved parenting skills. The treatment outcomes remained significant at 24-month follow-up when the intervention group was compared with the control group [ ]. However, no research exists about how this model works when the program is implemented in primary health care settings.
Regarding psychosocial interventions, there are 2 different aspects when converting these interventions from the research environment to the real world: dissemination and implementation of an innovation in clinical practice. Dissemination refers to how knowledge of the new practices is extended actively and passively, whereas implementation refers to the action of accommodating new practices into real treatment environments . Implementation gap refers to the difference between our knowledge of what works and how it works and the application of this knowledge in real-world practice. Research on the implementation of digital mental health interventions into routine care is scarce and studies on such interventions for preschool children are almost nonexistent. A consistent finding is that the vast majority of children with psychiatric problems early in life have unmet needs as only a minority of children with problems are referred and there are substantial delays in contacting specialist services [ ]. The great majority of adult psychiatric disorders begin in childhood or adolescence [ ]. Therefore, implementation of effective evidence-based treatments in real life plays a key role in child mental health service research [ ].
Previous implementation studies have emphasized that certain core implementation components and quality assurance measures ought to be included in the implementation strategy . Typically, the effects seen in the RCT settings decline during implementation [ ]. These quality measures are mostly related to the practitioner (referred to as coaches herein) of the program [ , ]. We ensured strict adherence to the same study protocol procedures as in the RCT study by accounting for certain quality measures as explained in the implementation plan (see Methods).
In this study, we implemented our aforementioned RCT study in primary health care settings to see how it would work in the real world. The first aim of this study was to compare certain child and family characteristics of the RCT intervention group with those of the first 882 families who received treatment during the primary care implementation phase. Both groups were based on population-based screening of 4-year-old children with high levels of disruptive behavior . We were particularly interested in finding out whether the level of child psychopathology, duration of problems, and impairment levels were similar when the RCT intervention group and the implementation group were compared. The second aim was to examine the differences between the 2 study groups in certain elements of the program, such as the time the parents spent reading psychoeducational material and completing skill exercises. The third aim was to compare the satisfaction levels in the 2 groups, namely, how the program affected their parenting skills, parental stress, and satisfaction with coaching.
We hypothesized that the screening procedure would also work in the implementation phase and no major differences would be found in the demographic factors or child psychopathology profiles when compared with the RCT study population. We also anticipated no major differences in the content of the program or parental satisfaction levels between the 2 groups.
Context of the Study
The study took place in Finland, a Nordic welfare state that provides its residents with public health services. Finnish child health clinics provide annual checkups that offer universal health care and are attended by 99.6% of children . The clinics try to identify problems that affect families with small children at an early stage and arrange for them to receive appropriate help. All parents are invited to bring their child to the child health clinic checkups about 15 times from birth to the age of 6 years. When the children attend the checkup at 4 years of age, they have reached a suitable stage in their development for identification of disruptive symptoms and provision of early support for families.
This study compared the intervention group from the RCT study with the first 882 families to receive parent training during the implementation phase. In both groups, all children attending child health clinics checkups at 4 years were screened.shows the flow chart of the study.
The complete RCT study protocol has previously been described in detail . In summary, the study design was an RCT with 2 parallel groups that were stratified by sex with 1:1 individual allocation. From 2011 to 2013, 5 municipalities in Southwest Finland joined the study. After the initial screening of 4656 children, 15.68 % (730/4656) 4-year-old children with high levels of disruptive behavioral problems were identified, and 464 (10.0%) parents were eligible to take part in the study and agreed. These parents were randomized into the SFSW intervention group (N=232) or an education control (N=232). The participants randomized to the intervention received an 11-session, internet-based parenting program that focused on skills for strengthening their parent-child relationships, together with a series of weekly telephone coaching sessions.
The implementation study group comprised the first 882 families to participate in the study following the success of the RCT. Initially, 12,780 children were screened, 1663 (13.01%) children with high levels of disruptive behavioral problems were identified, and 882 (6.9%) of the parents of 4-year-old children were eligible and agreed to take part in the study.
After an RCT study, when converting an evidence-based intervention into practice in a real-world setting, it is essential to tailor discrete multicomponent strategies for the implementation . Here, we identified the core implementation components of our implementation strategy. The following act as implementation drivers [ ]:
Recruitment, Staff Selection and Training
All coaches were professionals in health care (eg, public health nurses and nurses) and social services (children’s services). In our model, conduction and coaching of the intervention were centralized in the Research Centre for Child Psychiatry at the University of Turku. Each coach was trained on the protocol of the digital program by experienced Strongest Families clinicians. The training consisted of a theoretical background (such as preventive mental health and conduct behavior problems) and rehearsal phone calls. After the basic training, the coaches started the program by recruiting families. Then, they progressed to the closely monitored coaching calls (see below) with the actual families.
Before the utilization of the treatment model (screening and parent training intervention), a half-day training session was organized for the key actors in primary health care (such as public health nurses and doctors) to introduce the background of the program and treatment model and to make announcements and distribute other material.
In primary health care, the child health nurses were kept up to date on the progress of their clients within the Strongest Families program to ensure that they adhered to the parent training intervention model. This included summaries of the screening questionnaires complete with clinical outcome recommendations, updates on families consenting to the program, and a brief summary of the treatment program outcomes. In addition, monthly reports of questionnaire participation rates and numbers of completed treatments and dropouts were sent to child health clinic managers and chief doctors.
Ongoing Supervision and Staff Performance Evaluation
We conducted systematic, weekly supervision meetings with the individual coaches and group case conferences, where all coaches reviewed and discussed the families they were coaching. The more experienced coaches acted as supervisors. After each telephone coaching call, the coach assessed his or her own performance on a scale from 4 to 10. If self-assessment was greater than or equal to 6, the supervisor received a message from the digital platform and subsequently discussed the issue with the coach. About 10% of the recorded coaching calls were audited by the coach supervisor and evaluated for competency, with additional training and monitoring of future calls, when indicated.
Decision Supporting and Administration
The treatment model was introduced to the stakeholders of children and family services who decided the budget for the region or municipality. All parties signed the jointly funded annual research agreement on the treatment model implementation. Regular contact was maintained with the key actors of the region. The stakeholders received user-friendly monthly progress reports including the number of screened and enrolled families, etc. To integrate the model as a part of primary health care, regular training was conducted and information about the progress of the implementation study was distributed. Moreover, we involved local and national media to increase public awareness of our treatment model and more generally child mental health issues. Centralization of delivery of the digital parent training intervention and the development of the digital platform were the core of the facilitative administration. These strategies strengthened the implementation fidelity, which refers to the degree of adherence to implementation strategy and adaption process into local practices .
Of the original municipalities in the RCT study, the City of Turku and Town of Naantali, continued in the program when the implementation study began. Within the first year of the implementation, several municipalities and 2 provinces throughout Finland joined the study. The new treatment model had received publicity in media, and the RCT findings had been presented at national professional and scientific meetings.
The 882 families in the implementation phase came from throughout Finland, specifically the cities of Tampere, Espoo, Kouvola, Kuopio, and Turku; the towns of Lahti, Hyvinkää, Naantali, and Parainen; the municipalities of Kittilä, Mäntsälä, and Tuusula; and 3 provinces, South Karelia Social and Health Care District, Eksote, and Kainuu Social Welfare and Health Care Joint Authority.
A basic study plan was drafted by the research team for the implementation study, and this was introduced to senior child health clinic and health care officials from each of the health care districts. No major revisions were needed to accommodate regional needs.
A significant difference between the RCT and implementation study was that there was no control group, and the first 882 families that met the inclusion criteria were offered the chance to participate in the program. The child health nurses and their supervisors were key to introducing the early intervention program to the families.
Population-Based Screening Procedure and Recruitment
The screening procedure was similar in the RCT study and implementation phase, and it was integrated into the standard 4-year-old child health checkup visit. First, the supervising child health clinic staff and nurses in the participating areas were informed of the screening and intervention. Then, a 2-hour training session was held in each municipality to introduce the nurses to the background of the program and study plan and what they needed to do. We collected the data for all parents with children turning 4 years after January 2015 from the population register, and letters were mailed to parents in the middle of the month before the child’s fourth birthday. The letters included a short newsletter about the program, the form including questions of demographic information of parents and child, the Strengths and Difficulties Questionnaire (SDQ) [- ], and instructions for bringing the filled form to the annual checkup at the child health clinic. After the checkup, the nurses mailed the completed form to the research staff, and the answers were entered into the study database and scored with Access software (Microsoft Corp, Redmond, WA, USA). All the questionnaires were then mailed back to the respective child health clinics along with a separate SDQ score sheet, where they were reviewed together with the parents at their next visit to the child health clinic.
The inclusion and exclusion criteria for the implementation study were identical to the RCT trial. Parents were included in the study if their child scored 5 or more on the conduct subscale of the SDQ questionnaire and their answers to the SDQ impact supplement indicated that their child had behavioral problems. Parents were excluded if the child did not live with them because they were under the care of child protection services because of child custody, abuse, or neglect issues; the child did not speak in full sentences; or the parents had participated, or were participating, in other parent training or behavioral treatment. Children were also excluded if they met the diagnostic criteria for autism or a pervasive development disorder, Down syndrome, fetal alcohol syndrome, mental retardation, or a genetic diagnosis that would lead to mental retardation.
If the parents appeared to qualify for the study, they received a phone call from the program’s recruitment staff. They provided a brief introduction to the parent training program and reviewed the exclusion criteria. If the parents wanted to participate, they were directed to the study website to provide their formal consent. A form summarizing the recruitment call was mailed to the referring child health clinic to advise them of the outcome, as the municipalities were obliged to offer alternative treatment options for families not willing to participate in the parent training.
When the parents first logged into the study website, they were presented with the Web-based consent form for the study and a nonbinding program agreement, where the principles for successfully completing the program were outlined. These principles included adhering to the mutually agreed call schedule. The goal of the agreement was to ensure that the parents starting the program were reasonably committed to completing the training, which lasted for 3-4 months.
Parent Training Intervention
The SFSW intervention consisted of 2 components: the interactive website and the telephone coaching. The parent training program was delivered in a Web-based environment that the parent and coach both had access to, they had scheduled weekly phone calls, and they could also send additional messages via the website. No physical visits or face-to-face communication was conducted at any point during the program.
The program consisted of 11 weekly themes with associated phone coaching, and the program started with an introductory call from the coach after the parent had completed the baseline surveys. This provided an overview of the general objectives of the program and explained the learning methods and need for cooperation. It also set the initial goals for the program and introduced the first of the 11 weekly themes, which was called noticethegood. The goals were set in response to the problem behaviors that the child exhibited and that the program aimed to alleviate by teaching the parent how to use positive parenting skills and problem-solving abilities.
After the introduction, the parent was directed to the material for the first Web-based session, completed the associated skill training, and received the next coaching call a week later. The coaches used different verbal techniques during the phone calls, such as discussing different parenting models and role play as well as encouraging parents to practice every day. By using attributional questions, the coach could motivate parents to reflect on their own behavior as well as their child’s. The website tracked the parent’s activities on the site, and if they had not logged on for 2 days, the sites sent them a reminder, suggesting that they log in and complete the activities. The coach was notified if the parents had not been active on the website within 4 days of the last call. They then contacted the parent, encouraged to proceed with the program, and, if necessary, rescheduled the next call. The structured coach interaction with each client and monitoring are the key elements that distinguish coach-guided parent training from other forms of parent training, such as group-based, self-help, or email-assisted programs.
The content of each Web-based session was divided into the introduction, session content, video exercises, troubleshooting, review, and practical application of the new skills. Each section contained interactive and multimedia components, such as exercises and video clips. The parent was encouraged to complete the session by the next phone call. Some sessions also included supplementary material, which was emailed separately. The Web-based sessions followed the chronological order as shown in.
The sessions were conceptually divided into 3 sections: (1) basic positive parenting skills, (2) practical parenting skills, and (3) reinforcing acquired skills and sustained positive parenting. The aim of the basic positive parenting skills was to refocus the parent’s attention from the child’s problem behavior to noticing the child’s success in everyday life, as reflected in the name of the first session, notice the good. It also aimed to change the parent’s reactions to the child’s behavior from a negative to a positive response. Practical parenting skills focused on applying the basic skills in everyday situations, planning ahead with regard to daily activities, and using the supporting methods to reinforce positive behavior. These included sticker charts, when and then statements, and using time-out as a way to help the child and parent to regulate their emotions.
During the program, the parent learnt to solve problem situations using positive and practical skills and to understand their child better from a developmental and emotional perspective. The final sessions focused on reinforcing how they had applied the skills they had learnt, independent of the coaching support, and how they could sustain those skills beyond the active program.
|Session||Key training elements||Parental goals||Supplementary material|
|Notice the good||Positive and active parenting||Boosts child’s self-esteem; boosts parent’s self-esteem; and changes the parent’s view of their child||—a|
|Spread attention around||Positive, impartial parenting||Strengthens child’s empathy skills||—|
|Ignore whining and complaining||Positive, self-controlled parenting||Teaches parents self-regulation||—|
|Prepare for changes||Positive, proactive parenting||Reinforces good daily routines||—|
|Plan ahead at home||Positive, proactive parenting||Boosts child’s and parent’s self-esteem and involves the child in planning||—|
|Chart and stickers||Positive, active parenting||Involves the child in planning and reinforces good daily routines||Sticker chart and stickers|
|Plan ahead outside home||Positive, proactive parenting||Boosts child’s and parent’s self-esteem and involves the child in planning||—|
|Working with daycare||Positive cooperation and /communication between parent and day care||Helps child to manage and succeed||Daily report card|
|Time-out||Positive, self-controlled parenting||Teaches self-regulation and consistency||Digital timer|
|Problem solving, revision, and future application of skills||Positive daily parenting in future||Teaches parents skills to support child development and prepares for upcoming challenges||—|
|Booster||Skills review||Reminds parents of positive proactive parenting skills||Skills review chart|
Measurements Used in This Study
Demographic information of family and parents was obtained at the screening phase, and the variables that were included were information about the sex of the child, family structure, and the parents’ birth year and education and whether they had been unemployed. Child variables based on SDQ were collected from the form filled by parents at the screening phase.
Psychopathology was screened using the Finnish version of the SDQ for the parents of children aged 2 to 4 years , which is widely used as a screening and research tool and for clinical assessment and outcome evaluation. The SDQ consists of 25 items covering both positive and negative behaviors. These are divided into the 5 subscales of symptoms: emotional problems, conduct problems, hyperactivity or inattention, peer relationship problems, and prosocial behavior. Each subscale consists of 5 questions, and each item is rated on a scale with 3 possible answers: never (0 points), somewhat true (1 point), and certainly true (2 points). The SDQ impact supplement was also used, before inclusion in the program, to determine whether the child had a problem. If the parent said that they felt their child did have a problem, they were asked how chronic the problem was and about the distress, social impairment, and burden that the problem caused to others. The perceived difficulties were assessed with a single question: “Overall, do you think that your child has difficulties in one or more of the following areas: emotions, behavior or being able to get on with other people?” The alternatives were no, minor difficulties, definite difficulties, and severe difficulties.
Each participant’s time on the website was downloaded using appropriate time-out values, including the percentage of primary screens that the participants observed. The duration of therapeutic calls was downloaded and summarized at the end of the treatment. Data on treatment-related factors, program satisfaction, and therapeutic alliance questions were collected and provided to the research team at the end of treatment. The treatment-related factors covered 3 domains. The first was where the coaching took place (at home, at work, or in another place) and how often the coaching calls were received (every week/almost every week, a couple of times/once during the program, or never). The second was where the materials were read (at home, at work, or in another place), how often they were read (daily/almost daily, a couple of times a week/once a week, or never), and whom they read the material with (alone or with the child’s other parent). The third was the parents’ satisfaction with the program and therapeutic alliance questionnaire that explored their relationship with the program coach. These used a 5-point Likert scale (strongly disagree, disagree, not agree or disagree, agree, and strongly agree) and contained 13 propositions and 3 sections, covering the family program ( the program matched my expectations, I would recommend the program to my friends, if they were in need of similar help, in case I would need help in future, I would enroll to the program again, generally speaking, how content have you been with the Strongest Family program?), the effect of the program on their parenting skills ( I have learned skills, which have been helpful to me as a parent, I trust more my abilities to act as a parent, my relationship with my child has improved, my stress levels have been relieved), and the family coach ( the coach respected my views on parenting, the coach was professional, the coach encouraged problem solving, I could form a successful working relationship with the coach).
Ethical approval was received from the research ethics boards of the Hospital District of Southwest Finland and our Canadian program partners, the IWK Health Centre, Halifax, Nova Scotia, for the RCT study. We received ethical approval from the University of Turku for the implementation study.
To explore the differences between the implementation and the RCT intervention, we used the Pearson Chi-square test for categorical variables or the Fisher exact test, if the assumption of expected counts was violated. The assumption of equal variances and residuals normality was tested for continuous variables. If the assumptions were valid, the independent group t test was used. If normality assumptions failed, the t test was repeated after logarithmic transformation. If the log-transformed variable was not reaching assumptions, the Mann-Whitney U test with normal approximation for large samples (Z test) was applied. The effect sizes were estimated by the Cramér V for Chi-square test and Fisher exact test and by the Cohen d for t test and Z test.
The odds ratios (ORs) with 95% CI were estimated using logistic regression to examine the associations between discontinuation in the implementation and RCT groups. The statistical significance was classified by a 2-way P value of <.05. The statistical analyses were conducted using SAS statistical software version 9.4 (SAS Institute Inc, Cary, NC, USA).
The RCT group (N=464 including both intervention and control group) was screened from a population of 4656 children who were 4 years old (9.97%), whereas the implementation group including 882 children was screened from 12,780 children who were 4 years old (6.90%; χ12=45.0; P<.001). Of those 232 who were in the RCT intervention group, 52 (22.4%) discontinued the program before the tenth week, whereas the respective figure for those 882 in the implementation group was 109 (12.4%). The logistic regression indicated that parents in the implementation group were twice as likely to complete the program than in the RCT intervention group (OR 2.05, 95% CI 1.4-3.0; P<.001).
Family Demographics and Child Psychopathology
and shows the family demographic and child psychiatric information for the 882 children (553/882, 62.7% boys) in the implementation group and the 232 children (142/882, 61.2% boys) in the RCT intervention group. No differences were observed in family structure, parental age, or education level between the 2 groups. When the SDQ psychopathology of the children was compared, the groups had very similar scores. However, the implementation group had more moderate or severe difficulties (53.1%, 467/880; P=.02; effect size=−.071) and more severe impairment based on SDQ impact score (mean 1.1, SD 1.4; P=.006; effect size=0.214) than the RCT intervention group (44.4%, 103/232; mean 0.8, SD 1.2. It was notable that 42.0% (362/862) of the children in the implementation group and 35.4% (80/226) of the children in the RCT intervention had suffered from behavioral difficulties for more than a year before they were assessed.
As shown inand , most of the treatment variables were very similar in both groups. The telephone coaching calls were mostly received at home. Parents who took part in the RCT intervention phase were more likely to receive calls and read the material at work than the parents in the implementation group. Overall, the implementation group received an average of 11 calls (SD 2.2), whereas the RCT group received an average of 10 calls (SD 3.3; Z=3.41; P<.001) over the course of the program. The mean duration of the calls was very similar in both groups, lasting 37 to 38 min per call. The time spent on the website during the whole 11-week program was a mean of 451 (SD 174) min in the implementation group and 431 (SD 207) min in the RCT group (t318.8=1.83; P=.17).
|Family factors||Implementation, n (%)||Randomized controlled trial intervention, n (%)||Chi-square test (df)||P value||Effect sizea|
|Family and parent variables|
|Sex of the child||0.2 (1)||.68||0.013|
|Male||553 (62.7)||142 (61.2)|
|Female||329 (37.3)||90 (38.8)|
|Family structure||0.3 (1)||.58||−.017|
|Two biological parents||720 (81.9)||192 (83.5)|
|Other||159 (18.1)||38 (16.5)|
|Maternal age at childbirth||0.4 (1)||.55||0.018|
|Up to 24 years||112 (12.8)||26 (11.3)|
|25 plus years||764 (87.2)||204 (88.7)|
|Paternal age at childbirth||2.4 (1)||.12||0.048|
|Up to 24 years||69 (8.2)||11 (5.1)|
|25 plus years||778 (91.9)||207 (95.0)|
|Maternal education||1.0 (1)||.31||−.031|
|Comprehensive school or lower or secondary education||339 (38.9)||98 (42.6)|
|College/university degree||532 (61.1)||132 (57.4)|
|Paternal education||0.1 (1)||.71||0.012|
|Comprehensive school or lower or secondary education||445 (54.7)||115 (53.2)|
|College/university degree||369 (45.3)||101 (46.8)|
|Child variables based on Strengths and Difficulties Questionnaire|
|Minor difficulties||413 (46.9)||129 (55.6)|
|Moderate or severe difficulties||467 (53.1)||103 (44.4)|
|Length of difficulties||3.2 (1)||.07||− .055|
|Less than 1 year||500 (58.0)||146 (64.6)|
|1 year or more||362 (42.0)||80 (35.4)|
aThe effect size is measured by Cramér V.
|Family factors||Implementation, mean (SD)||Randomized controlled trial intervention, mean (SD)||Z-test||t test (df)||P value||Effect sizea|
|Total score||15.5 (4.6)||14.9 (4.4)||—||1.76 (1108)||.08a||0.139b|
|Emotional score||1.9 (1.8)||1.7 (1.5)||−1.1||—||.29b||0.128|
|Peer problems score||2.4 (1.7)||2.3 (1.6)||—||1.16 (1109)||.25a||0.081|
|Prosocial behavior||5.9 (1.9)||6.0 (1.8)||—||−0.80 (1112)||.43||−.059|
|Hyperactive score||4.9 (2.5)||4.7 (2.3)||—||1.14 (1111)||.26||0.086|
|Conduct score||6.2 (1.3)||6.2 (1.3)||—||0.49 (1112)||.62a||0.042|
|Impact score||1.1 (1.4)||0.8 (1.2)||−2.8||—||.006b||0.214|
aThe P value after logarithmic transformation by t test.
bThe P value was determined by the Mann-Whitney U test with normal approximation for large samples.
|Treatment-related factors||Implementation, n (%)||Randomized controlled trial intervention, n (%)||Chi-square test (df)||Fisher exact test||P value||Effect sizea|
|Where calls were received?|
|At home||2.6 (2)||—b||.27||.055|
|Every week/almost every week||563 (76.2)||125 (82.2)|
|Couple times/once during the program||148 (20.0)||23 (15.1)|
|Never||28 (3.8)||4 (2.6)|
|At work||5.8 (2)||—||.054||.082|
|Every week/almost every week||118 (16.0)||31 (23.3)|
|Couple times/once during the program||146 (19.8)||30 (22.6)|
|Never||475 (64.3)||72 (54.1)|
|Other place than home or work||3.1 (2)||—||.21||.60|
|Every week/almost every week||54 (7.3)||7 (5.7)|
|Couple times/once during the program||306 (41.4)||42 (34.4)|
|Never||379 (51.3)||73 (59.8)|
|Where materials were read?|
|At home||0.5 (2)||—||.79||.023|
|Daily/almost daily||50 (6.8)||11 (7.0)|
|Couple times a week/once a week||680 (92.0)||144 (91.1)|
|Never||9 (1.2)||3 (1.9)|
|Daily/almost daily||2 (0.3)||1 (0.7)|
|Couple times a week/once a week||193 (26.1)||57 (39.9)|
|Never||544 (73.6)||85 (59.4)|
|Other place than home or work||4.2 (2)||—||.12||.070|
|Daily/almost daily||7 (1.0)||1 (0.8)|
|Couple times a week/once a week||152 (20.6)||16 (12.8)|
|Never||580 (78.5)||108 (86.4)|
|Whom were materials read with?|
|Daily/almost daily||41 (5.6)||10 (6.4)|
|Couple times a week/once a week||661 (89.5)||141 (89.8)|
|Never||37 (5.0)||6 (3.8)|
|With the other parent of the child||0.7 (2)||—||.69||.029|
|Daily/almost daily||10 (1.4)||2 (1.4)|
|Couple times a week/once a week||423 (57.2)||78 (53.4)|
|Never||306 (41.4)||66 (45.2)|
|Completed part of the program||24.3 (3)||—||<.001||.148|
|Only introduction part||16 (1.8)||14 (6.0)|
|Basic skills (weeks 1-4)||51 (5.8)||18 (7.8)|
|Functional skills (weeks 5-9)||45 (5.1)||24 (10.3)|
|Revision and future application of skills (weeks 10-11)||770 (87.3)||176 (75.9)|
aThe effect size is measured by Cramér V.
|Treatment-related factors||Implementation||Randomized controlled trial intervention||Z-test||t test (df)||P valuea||Effect sizeb|
|Mean (SD)||Range||Mean (SD)||Range|
|Number of calls||11 (2.2)||2-14||10 (3.3)||1-14||3.41||—c||<.001||.146|
|Duration of call (min)||38 (10.3)||3-102||37 (13.5)||2-82||—||0.26 (279.5)||.79||.021|
|Duration of sign-ins (by staying on the website, min)||451 (174)||72-1330||431 (207)||29-1362||—||1.38 (318.8)||.17||.107|
aThe P value was determined by Mann-Whitney U test with normal approximation for large samples.
bThe effect size is measured by Cohen d.
Asshows, the participants in both groups reported high levels of satisfaction with how the program had improved their parental skills. The satisfaction questionnaire was filled by 744 of 882 participants (84.4%) in the implementation and 162 of 232 participants (69.8%) in the RCT intervention phase. In the implementation group, 83.4% (619/742) to 98.5% (731/742) reported high satisfaction with various aspects of the program, and in the RCT group, 82.7% (134/162) to 98.2% (159/162) were very highly satisfied. When they were asked about the overall impact of the program, 88.4% (658/744) of the parents in the implementation groups said it had matched their expectations and 94.0% (699/744) said it had matched their needs. The respective figures for the RCT intervention group were 82.7% (134/162) and 93.8% (152/162). In addition, 77.1% (498/742) of implementation group parents and 64.8% (105/162) of the RCT parents reported that the program had reduced their stress. Finally, both groups reported very high levels of satisfaction (96.8% [719/743] and 100% [162/162], respectively) with the skills and professionalism of the telephone coaches and their relationships with them. We conducted a sensitivity analysis on the satisfaction-related factors, where we excluded 2 municipalities from the implementation sample that participated in the RCT study (altogether 159 cases). The results were not qualitatively different from the tests on the complete data.
|Satisfaction-related factorsa||Implementation, n (%)||Randomized controlled trial intervention, n (%)||Chi-square test (df)||Fisher exact test||P value||Effect sizeb|
|Overall impact: family program|
|The program matched my expectations||3.96 (1)||—c||.047||−.066|
|Disagree/neutral||86 (11.6)||28 (17.3)|
|Agree||658 (88.4)||134 (82.7)|
|The program met my needs||0.004 (1)||—||.95||−.002|
|Disagree/neutral||45 (6.1)||10 (6.2)|
|Agree||699 (94.0)||152 (93.8)|
|I would recommend the program to my friends, if they were in need of similar help||1.01 (1)||—||.31||.033|
|Disagree/neutral||25 (3.4)||3 (1.9)|
|Agree||719 (96.6)||159 (98.2)|
|In case I would need help in the future, I would enroll to the program again||0.16 (1)||—||.69||−.013|
|Disagree/neutral||97 (13.0)||23 (14.2)|
|Agree||647 (87.0)||139 (85.8)|
|Generally speaking, how content have you been with the Strongest Family program?||—||0.20||.18||.020|
|Disagree/neutral||20 (2.7)||3 (1.9)|
|Agree||724 (97.3)||159 (98.2)|
|Overall impact: effects on parenting skills|
|I have learned skills, which have been helpful to me as a parent||—||0.09||.18||−.047|
|Disagree/neutral||11 (1.5)||5 (3.1)|
|Agree||731 (98.5)||157 (96.9)|
|I trust more in my abilities to act as a parent||0.39 (1)||—||.53||.021|
|Disagree/neutral||86 (11.6)||16 (9.9)|
|Agree||656 (88.4)||146 (90.1)|
|My relationship with my child has improved||0.03 (1)||—||.87||.006|
|Disagree/neutral||123 (16.6)||26 (16.1)|
|Agree||619 (83.4)||136 (84.0)|
|My stress levels have been relieved||0.31 (1)||—||.57||−.019|
|Disagree/neutral||244 (32.9)||57 (35.2)|
|Agree||498 (77.1)||105 (64.8)|
|Direct impact: family coach|
|The coach respected my views on parenting||—||0.04||.09||.063|
|Disagree/neutral||16 (2.2)||0 (0)|
|Agree||726 (97.8)||162 (100.0)|
|The coach was professional||—||0.16||.40||.034|
|Disagree/neutral||19 (2.6)||2 (1.2)|
|Agree||723 (97.4)||160 (98.8)|
|The coach encouraged problem solving||—||0.19||.55||.028|
|Disagree/neutral||17 (2.3)||2 (1.2)|
|Agree||725 (97.7)||160 (98.8)|
|I could form a successful working relationship with the coach||—||0.04||.07||.067|
|Disagree/neutral||24 (1.2)||1 (0.6)|
|Agree||719 (96.8)||161 (99.4)|
aDisagree/neutral combines strongly disagree, disagree, and not agree or disagree. Agree combines agree and strongly agree.
bThe effect sizes are measured by Cramér V.
This study describes the content and process of implementing a Web-based parent training program with telephone coaching in comparison with the RCT intervention. The program is unique because it is based on screening children from the general population during routine child health clinic checkups at the age of 4 years. First, the findings show that the characteristics of families recruited to the RCT study and the implementation groups were very similar. Second, the duration and content of the Web-based training and phone coaching were similar in both groups. Third, the satisfaction rates did not differ between the groups. Finally, against expectations, the discontinuation rate of the program was higher in the RCT group than in the implementation group.
The screening procedures that were used to identify children at risk during both the RCT and implementation phases resulted in similar profiles for families and child psychopathology. The only exceptions to the profiles were that the implementation group had 2-fold more moderate or severe difficulties (53,1%, 467/880) and more severe impairment based on SDQ impact score than the RCT intervention group (44.4%, 103/232). This result may indicate that the parents of the children with more severe disruptive behavior are more motivated to seek help and to keep up with the program than those of the children with minor behavioral problems. The same outcome has been found previously . We suggest that the parental motivation related to the volume of the problem may act as one of the implementation drivers.
The study documented the duration and content of the 2 key elements of the remote parent training program, which were Web-based training and phone coaching. Both sets of parents spent about 80 min engaged in the program each week, when the figures for website use and coaching were combined. However, this did not include the time parents spent practicing the positive training skills with the child during the program, which was the key goal of the Web-based content and phone coaching.
The study showed high and similar parent satisfaction rates in both groups. The level of satisfaction with the program, how the program affected parenting skills, and how the parents worked with the phone coaches remained very high when the program made the transition from the RCT to the implementation phase. Most of the parents felt they had been able to form a successful working relationship with the coach.
Finally, the study shows that less than 12,7 (112/882) of parents discontinued the program before the tenth week in the implementation group, which was almost 2-fold lower (24,1%, 56/232) than that in the RCT study. This indicates that the Web-based parent training programs supplemented with telephone coaching may result in high success. In our study, we identified core implementation drivers to facilitate the implementation process. The intervention fidelity was ensured in the implementation phase, by centralization of the conduction and coaching of the intervention, systematic quality assurance enabled via a digital platform, and supervision. Moreover, both ongoing training of the primary health care staff and stakeholders and facilitative administration, that is, regular meetings, user-friendly reports, and involving media, were likely to lead to successful implementation.
To our knowledge, the SFSW intervention is the first Web-based parent training program to use population-based screening to select its subjects and provide telephone coaching. Other digital interventions have reported using automated messages or emails to interact with parents [, ]. During the program, the coach and the parent formed a working relationship, which was crucial for meeting mutually agreed goals [ , ]. Telephone coaching has a number of benefits: it enables real-time problem solving with the parent, provides direct feedback on how the parents adopt strategies and skills, and motivates the parent to continue the learning process. In addition, the website tracking and telephone coaching worked together to ensure sufficient exercises were conducted by the parent between the weekly sessions.
The findings of this study are important in a number of ways. They can help to inform best practice in implementing interventions that target large numbers of parents in a wide geographical area. They can also help to tackle the problems that are inherent with traditional parent training group interventions, such as drop-out rates and practical and resource issues. We have demonstrated that it is possible to replicate the success of an RCT at the implementation stage if the families are motivated and the structure of the program provides the right elements and meets their needs. Our experience also underlines that the ongoing supervision of those delivering the coaching element of the training is essential. It is notable that the use and adherence rates of digital mental health interventions might be lower in real-world settings, as implemented outside of research settings . Our findings show that discontinuation of the intervention in the implementation group was exceptionally low and users were highly satisfied with the intervention.
Web-based interventions provide various opportunities for extending services to people with mental health  and behavioral problems, as they provide an effective way of reaching people, including the parents of children, as with our program. Moving interventions outside traditional clinics and into people’s daily environments, like the internet, can facilitate better access to mental health services [ ]. Web-based interventions can also remove the barriers associated with the face-to-face interventions and enable people to seek help for mental health problems without fear of being stigmatized [ , , ]. From the families’ point of view, digitally delivered programs can offer faster and more flexible services without the need for transport, juggling work schedules, arranging childcare, or the practical cost of accessing services. On the basis of the evidence, Web-based telephone-assisted interventions for children with disruptive behavioral problems could play a significant role in parent training by engaging families into low-threshold primary care services to promote the mental well-being of families and children [ , , ].
As nearly all Finnish families use the child health clinic services, our treatment model reaches a very high proportion of the population. This enables early access to the low-threshold intervention, which is in accordance with the core principles of the Finnish child health clinic system, that is, early prevention, early detection, and increasing the awareness of physical and mental health. The elements of the treatment model constitute a new form of preventive intervention for disruptive behavior.
There were some limitations to our study. We did not have any information about changes in the children’s psychiatric problems or changes in parenting skills in the implementation group, which would have told us if the real-life implementation was as effective as the RCT study setting. We only had information about the time they spent on the website and telephone coaching, their satisfaction with the program, and how well they felt it met their needs. It must be noted that neither the therapeutic alliance nor the parent satisfaction questionnaire was validated for the Finnish settings. The alliance questionnaire was based on the original WAI (Working Alliance Inventory), whereas the satisfaction questionnaire was self-composed. Both the Web-based element and the coaching calls encouraged the parents to practice the skills they learnt in real life, but we do not have information about how much the parents actually did this. Information from public health nurses would have helped us to evaluate how the intervention was integrated into the primary health context. Finally, it is important to note that this study is an implementation study and not a sustainability study. However, the implementation sample in this study has been gathered up to 3 years after the RCT study ended. Sustaining an intervention in practice over time is different from scaling up interventions in practice. To maintain the effects observed in this study over a longer period may require additional strategies.
This study has important implications for planning low-cost, low-threshold, early, population-based, and evidence-based interventions in future. Our study revealed very similar findings between the RCT and implementation groups with regard to the profiles of the families and child psychopathology, the duration and content of the Web-based training and phone coaching, and the parents’ satisfaction rates. The discontinuation rate was lower and disruptive behavior was more severe in the implementation group. These findings may suggest that the parents of the children with more severe disruptive behavior are highly motivated to receive help and complete the program. In this sense, parental motivation may be seen as an implementation driver. To our understanding, the successful implementation in this study constituted of numerous factors and their seamless interaction: the convenient and easy-to-use Web-based format with supplementary telephone coaching by trained professionals, sufficient training for stakeholders and primary health staff on the procedures, the psychosocial evaluation, and the support that the program families would receive. Moreover, we suppose that the central role is played by the primary health care, which in Finnish child health clinics has a potential to reach almost 100% of the children. We believe that this program provides a number of advantages over traditional parent training models, which face enormous challenges such as practical, resource, and retention issues, and that it could provide a blueprint for other cost-effective preventive and early intervention child mental health programs.
The research was funded by the Strategic Research Council of the Academy of Finland (Decision No. 303581), the Academy of Finland (Decision No. 320162), the Canadian Institutes of Health Research, the Jane and Aatos Erkko Foundation, the Association of Friends of the University Children’s Hospitals, the Turku University Hospital Expert Responsibility Area Research Funding, and a Canada Research Chair.
Conflicts of Interest
- Dretzke J, Frew E, Davenport C, Barlow J, Stewart-Brown S, Sandercock J, et al. The effectiveness and cost-effectiveness of parent training/education programmes for the treatment of conduct disorder, including oppositional defiant disorder, in children. Health Technol Assess 2005 Dec;9(50):iii, ix-x, 1-233 [FREE Full text] [Medline]
- Dretzke J, Davenport C, Frew E, Barlow J, Stewart-Brown S, Bayliss S, et al. The clinical effectiveness of different parenting programmes for children with conduct problems: a systematic review of randomised controlled trials. Child Adolesc Psychiatry Ment Health 2009 Mar 04;3(1):7 [FREE Full text] [CrossRef] [Medline]
- Eyberg SM, Nelson MM, Boggs SR. Evidence-based psychosocial treatments for children and adolescents with disruptive behavior. J Clin Child Adolesc Psychol 2008 Jan;37(1):215-237. [CrossRef] [Medline]
- Perrin EC, Leslie LK, Boat T. Parenting as primary prevention. JAMA Pediatr 2016 Dec 01;170(7):637-638. [CrossRef] [Medline]
- Jensen PS, Goldman E, Offord D, Costello EJ, Friedman R, Huff B, et al. Overlooked and underserved: "action signs" for identifying children with unmet mental health needs. Pediatrics 2011 Nov;128(5):970-979 [FREE Full text] [CrossRef] [Medline]
- Kazdin AE, Blase SL. Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspect Psychol Sci 2011 Jan;6(1):21-37. [CrossRef] [Medline]
- Kazdin AE, Rabbitt SM. Novel models for delivering mental health services and reducing the burdens of mental illness. Clin Psychol Sci 2013 Jan 23;1(2):170-191. [CrossRef]
- Breitenstein SM, Gross D, Christophersen R. Digital delivery methods of parenting training interventions: a systematic review. Worldviews Evid Based Nurs 2014 Jun;11(3):168-176. [CrossRef] [Medline]
- Barak A, Grohol JM. Current and future trends in internet-supported mental health interventions. J Technol Hum Serv 2011 Jul;29(3):155-196. [CrossRef]
- Taylor TK, Webster-Stratton C, Feil EG, Broadbent B, Widdop CS, Severson HH. Computer-based intervention with coaching: an example using the Incredible Years program. Cogn Behav Ther 2008;37(4):233-246 [FREE Full text] [CrossRef] [Medline]
- Hammond GC, Croudace TJ, Radhakrishnan M, Lafortune L, Watson A, McMillan-Shields F, et al. Comparative effectiveness of cognitive therapies delivered face-to-face or over the telephone: an observational study using propensity methods. PLoS One 2012;7(9):e42916 [FREE Full text] [CrossRef] [Medline]
- Webster-Stratton C, Kolpacoff M, Hollinsworth T. Self-administered videotape therapy for families with conduct-problem children: comparison with two cost-effective treatments and a control group. J Consult Clin Psychol 1988 Aug;56(4):558-566. [Medline]
- Official Statistics of Finland (OSF). Statistics Finland. Helsinki: Statistics Finland; 2017. Use of information and communications technology by individuals URL: https://www.stat.fi/til/sutivi/index_en.html [accessed 2018-02-07] [WebCite Cache]
- Baumel A, Pawar A, Kane JM, Correll CU. Digital parent training for children with disruptive behaviors: systematic review and meta-analysis of randomized trials. J Child Adolesc Psychopharmacol 2016 Oct;26(8):740-749. [CrossRef] [Medline]
- Sourander A, McGrath PJ, Ristkari T, Cunningham C, Huttunen J, Lingley-Pottie P, et al. Internet-assisted parent training intervention for disruptive behavior in 4-year-old children: a randomized clinical trial. JAMA Psychiatry 2016 Apr;73(4):378-387. [CrossRef] [Medline]
- Sourander A, McGrath PJ, Ristkari T, Cunningham C, Huttunen J, Hinkka-Yli-Salomäki S, et al. Two-year follow-up of internet and telephone assisted parent training for disruptive behavior at age 4. J Am Acad Child Adolesc Psychiatry 2018 Sep;57(9):658-668.e1. [CrossRef] [Medline]
- May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci 2016 Dec 19;11(1):141 [FREE Full text] [CrossRef] [Medline]
- Gyllenberg D, Gissler M, Malm H, Artama M, Hinkka-Yli-Salomäki S, Brown AS, et al. Specialized service use for psychiatric and neurodevelopmental disorders by age 14 in Finland. Psychiatr Serv 2014 Mar 01;65(3):367-373 [FREE Full text] [CrossRef] [Medline]
- World Health Organization. 2018. Adolescents: health risks and solutions URL: http://www.who.int/en/news-room/fact-sheets/detail/adolescents-health-risks-and-solutions [accessed 2018-06-21] [WebCite Cache]
- Michelson D, Davenport C, Dretzke J, Barlow J, Day C. Do evidence-based interventions work when tested in the. Clin Child Fam Psychol Rev 2013 Mar;16(1):18-34. [CrossRef] [Medline]
- Fixsen DL, Blase KA, Naoom SF, Wallace F. Core implementation components. Res Soc Work Pract 2009 May 27;19(5):531-540. [CrossRef]
- Dobson KS. Definitional and practical issues in the assessment of treatment integrity. Clin Psychol: Sci Pr 2005 Dec 01;12(4):384-387. [CrossRef]
- Sánchez V, Steckler A, Nitirat P, Hallfors D, Cho H, Brodish P. Fidelity of implementation in a treatment effectiveness trial of reconnecting youth. Health Educ Res 2007 Feb;22(1):95-107. [CrossRef] [Medline]
- Fossum S, Ristkari T, Cunningham C, McGrath PJ, Suominen A, Huttunen J, et al. Parental and child factors associated with participation in a randomised control trial of an internet-assisted parent training programme. Child Adolesc Ment Health 2016 Nov 10;23(2):71-77. [CrossRef]
- National Institute for Health and Welfare. Department of Health and Welfare. Child welfare clinic URL: https://www.thl.fi/en/web/lapset-nuoret-ja-perheet/peruspalvelut/aitiys_ja_lastenneuvola/lastenneuvola [accessed 2018-02-07] [WebCite Cache]
- McGrath PJ, Sourander A, Lingley-Pottie P, Ristkari T, Cunningham C, Huttunen J, et al. Remote population-based intervention for disruptive behavior at age four: study protocol for a randomized trial of Internet-assisted parent training (Strongest Families Finland-Canada). BMC Public Health 2013 Oct 21;13:985 [FREE Full text] [CrossRef] [Medline]
- Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci 2015 Feb 12;10:21 [FREE Full text] [CrossRef] [Medline]
- Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, StaRI Group. Standards for Reporting Implementation Studies (StaRI): explanation and elaboration document. BMJ Open 2017 Dec 03;7(4):e013318 [FREE Full text] [CrossRef] [Medline]
- Goodman R. The Strengths and Difficulties Questionnaire: a research note. J Child Psychol Psychiatry 1997 Jul;38(5):581-586. [Medline]
- Koskelainen M, Sourander A, Kaljonen A. The Strengths and Difficulties Questionnaire among Finnish school-aged children and adolescents. Eur Child Adolesc Psychiatry 2000 Dec;9(4):277-284. [Medline]
- Borg A, Kaukonen P, Salmelin R, Joukamaa M, Tamminen T. Reliability of the strengths and difficulties questionnaire among Finnish 4-9-year-old children. Nord J Psychiatry 2012 Dec;66(6):403-413. [CrossRef] [Medline]
- Lingely-Pottie P, McGrath PJ. A therapeutic alliance can exist without face-to-face contact. J Telemed Telecare 2006;12(8):396-399. [CrossRef] [Medline]
- Lingley-Pottie P, McGrath PJ. Distance therapeutic alliance: the participant's experience. ANS Adv Nurs Sci 2007 Dec;30(4):353-366. [CrossRef] [Medline]
- Fleming T, Bavin L, Lucassen M, Stasiak K, Hopkins S, Merry S. Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. J Med Internet Res 2018 Jun 06;20(6):e199 [FREE Full text] [CrossRef] [Medline]
- Bee PE, Bower P, Lovell K, Gilbody S, Richards D, Gask L, et al. Psychotherapy mediated by remote communication technologies: a meta-analytic review. BMC Psychiatry 2008 Jul 22;8:60 [FREE Full text] [CrossRef] [Medline]
- Slone NC, Reese RJ, McClellan MJ. Telepsychology outcome research with children and adolescents: a review of the literature. Psychol Serv 2012 Aug;9(3):272-292. [CrossRef] [Medline]
|OR: odds ratio|
|RCT: randomized controlled trial|
|SDQ: Strengths and Difficulties Questionnaire|
|SFSW: Strongest Families Smart Website|
Edited by G Eysenbach; submitted 29.06.18; peer-reviewed by F Drozd, J Doty; comments to author 16.08.18; revised version received 09.11.18; accepted 31.12.18; published 11.04.19
©Terja Ristkari, Marjo Kurki, Auli Suominen, Sonja Gilbert, Atte Sinokki, Malin Kinnunen, Jukka Huttunen, Patrick McGrath, Andre Sourander. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 11.04.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.