Patient adherence, sometimes also termed compliance, is most broadly defined as “the extent to which the patient’s behavior (in terms of taking medications, following diets, or executing other lifestyle changes) coincides with medical or health advice” [1]. Within the field of psychotherapy, adherence is usually conceptualized within the therapeutic tradition or according to the target condition that is being studied [2]. Accordingly, adherence is measured in various ways, for example, as qualitative or quantitative aspects of homework compliance [3], in-session engagement [4], or patient-therapist alliance [5]. Absence of patient adherence, on the other hand, is often referred to as resistance or ambivalence [6,7], and there are various therapeutic strategies that are specifically designed to target such resistance or ambivalence, such as motivational interviewing [8].

One important aspect of patient adherence is the logical assumption that adherent patients will have better treatment outcomes than nonadherent patients, though the evidence has not always supported this assumption. There is, for example, evidence that adherence to homework assignments in cognitive behavioral therapy (CBT) is associated with better outcomes, with a small to large effect of homework quality and quantity, but with substantial heterogeneity between studies [3,9]. Similarly, adherence to exposure and response prevention homework has been shown to be a strong predictor of short- and long-term outcome in CBT for obsessive-compulsive disorder (OCD) [10,11]. However, results from the field of pediatric anxiety have failed to establish a clear relationship between homework adherence and treatment outcome [12]. For example, in 1 study, neither the number of attended CBT sessions nor completed homework assignments predicted treatment outcome but a 1-item clinician-rated adherence measure did [13]. One study of CBT for anxiety disorders applied a combined measure of adherence, consisting of a weekly clinician rating of homework completion and a rating of engagement in the therapy session. The authors found that a combination of high treatment dose (high session attendance and completed exposure exercises) and high engagement predicted better treatment outcomes [14]. Importantly, it has been suggested that the adherence-outcome relationship could be affected by the quality of the adherence measure, and thus, reliable and valid measures are essential as research and clinical tools [15].

Internet-delivered behavioral interventions have emerged as an interdisciplinary development of psychotherapy and modern information technology. Internet-delivered behavioral interventions share common features as they mirror face-to-face psychotherapy regarding content and therapeutic techniques but are presented online via a personal login to a website or internet portal. The format usually resembles an online course, with instructive texts, pictures, videos, and written exercises, with the content structured similar to a book with chapters or modules. When therapist contact and feedback are involved, the treatment is referred to as guided or clinician-supported and otherwise as unguided or self-help [16]. Internet-delivered CBT (ICBT) has been one of the most prominent formats of this novel branch of interventions, with over 100 randomized controlled trials (RCTs) conducted in adults [17] and currently 19 RCTs in children and adolescents [18].

Adherence may be particularly difficult to measure in internet-delivered behavioral interventions [19]. In face-to-face treatment, many of the proposed active treatment components, such as behavioral or cognitive strategies, are conducted in-session together with a clinician. In internet-delivered behavioral interventions, the key therapy components are delivered online and practiced without the physical presence of the clinician. Depending on the intervention, the engagement in the active treatment components of the intervention may have low correlation with the number of completed chapters. For instance, patients may go on practicing their homework assignments without necessarily needing to log into the system periodically. Therefore, the number of completed chapters in internet-delivered behavioral interventions is not comparable with the number of completed sessions in face-to-face interventions and, thus, may not be a reliable measure of treatment adherence. As pointed out by Sieverink et al [20], the concept of “intended usage” could be a helpful extension of the traditional definition of adherence. Intended usage is defined as “the extent to which individuals  should experience the content (of the intervention) to derive maximum benefit from the intervention, as defined or implied by its creators” [21].

Thus, defining and measuring the core therapeutic concepts and minimal effective use of those strategies may be a vital aspect of any measure of adherence to internet-delivered interventions. However, less than half of the studies in the field apply such a definition, and only few provide a justifiable definition of minimal required usage [20]. Currently, the most prevalent operationalizations of adherence are number of treatment modules completed by patients and number of logins [20,22]. The relationship between adherence and clinical outcome has thus far been limited by inconsistent reporting of adherence and a lack of agreement on measures [18,22].

In summary, there is an important research gap and a need for reliable and valid measures of patient adherence to internet-delivered interventions. We take the perspective that a useful measure of adherence to internet-delivered behavioral interventions should (1) not only capture the structural aspects of the internet format (such as completed chapters or modules) but also consider the intended usage of its central therapeutic aspects (such as use of behavioral strategies) and (2) include relevant, observable behaviors that represent the patients’ adherence in a way that guides clinically meaningful decisions. Thus, the aim of this project was to construct and validate an adherence measure, evaluate its psychometric properties, and establish whether this new measure correlates better with treatment outcome than other traditional measures of adherence (eg, number of logins and completed chapters).

A novel, clinician-rated measure called internet intervention Patient Adherence Scale (iiPAS) was constructed in a 4-step process following recommendations from the psychometric literature [23].

All interventions were provided by researchers and clinicians associated to the BiP network, a collaboration between the Karolinska Institutet and the Child and Adolescent Mental Health Service in Stockholm, with a main research focus on development and evaluation of internet-delivered interventions for childhood mental disorders. All interventions in the current evaluation were clinician-guided ICBT interventions that also included parent support. Clinician advice, support, and feedback were given predominantly via written messages within the secure internet platform or via occasional telephone calls. For a short summary of the interventions, see Table 1. More detailed descriptions can be found in the original publications [29-33].

The included clinical trials applied similar but not identical procedures. A description of the common aspects of the procedures is provided here; for a detailed description, see the respective trial publications. Participants in patient samples 1 and 2 were recruited via clinician- or self-referrals. Selection of eligible participants included both telephone screenings with a clinician and an in-person assessment with the child or adolescent and at least one caregiver. Fulfillment of diagnostic criteria was established by a clinician (most often a clinical psychologist) using the Mini-International Neuropsychiatric Interview child version (MINI Kid [34]) or the Anxiety Disorder Interviews Schedule Child and Parent versions (ADIS C/P [26]). Decision about inclusion or exclusion of participants was made after the in-person assessment, followed by treatment allocation.

The BiP TIC and BiP Anxiety trials had a randomized controlled design [24,29]. In BiP TIC, participants were randomized to either habit reversal training or exposure and response prevention. In BiP Anxiety, participants were randomized to either ICBT or an active control condition. The remaining trials had an open, within-group design.

The clinical outcome measures were assessed at pre- and posttreatment. Activity data from the use of the internet platform (number of logins, written characters, login time duration, and completed treatment chapters) were automatically collected during the treatments. The iiPAS was administered by the clinicians participating in the trials, on 2 separate occasions: halfway through the treatment (iiPAS-mid) and at posttreatment (iiPAS-post).

An initial version of the iiPAS was administered to 50 children and adolescents, aged between 7 and 17 years, receiving ICBT for OCD or anxiety disorders. Preliminary analyses of Cronbach alpha indicated excellent internal consistency (alpha=.93).

The objective of this project was to develop and psychometrically evaluate a novel measure of patient adherence in internet-delivered behavioral interventions. The measure was designed in close collaboration with experienced clinicians and researchers within the field and resulted in a clinician-rated, 5-item scale that was tested in a sample of children and adolescents with various behavioral disorders. Descriptive statistics indicated an appropriate range and variability of the iiPAS. The scale demonstrated excellent internal consistency. Consistently, factor analyses showed that the iiPAS measures one underlying factor, in agreement with the theoretical assumption that patient adherence within the context of internet interventions can be understood as one homogeneous construct.

Objective behavioral data were available through automatically registered internet platform data of number of logins, logged in time, written characters, and percentage of completed chapters. The iiPAS was significantly associated with those objective variables, apart from logged in time, which strongly suggests that the iiPAS indeed measures key aspects of patient adherence. The iiPAS was also significantly associated with symptom severity changes after treatment. Post hoc analyses revealed that 2 of the 5 items at midtreatment, engagement and motivation to change, were particularly associated with a decrease in symptoms. The midtreatment administration of the iiPAS may be especially interesting from a clinical point of view, as half of the internet treatment is still ahead. For example, a low iiPAS midtreatment score could inform the clinician that the adherence of the patient should be addressed for optimal outcome. The strong correlation of the iiPAS-mid and iiPAS-post suggests that little change in adherence is to be expected in the second half of the treatment. This could mean that other interventions such as motivational interventions might be needed to increase adherence, or that the intervention might need to be intensified or augmented with telephone, video calls or in-person therapist support.

Interestingly, none of the objective platform usage variables were associated with symptom severity change at posttreatment. This could possibly indicate that the mere adherence to the formal structure of an internet intervention (logging in, writing text, and completing chapters) is less important for the clinical outcome than actively engaging with the therapeutic content that is presented in the intervention (such as testing new behavioral or cognitive strategies and delivering homework exercises with high quality). The existing literature on patient adherence supports that quality of homework exercise completion is a better predictor of treatment outcome than number of completed exercises [10,13]. However, studies that are specifically designed to clarify the importance of adhering to the formal structure of a treatment in comparison with adhering to the therapeutic content are needed.

Our analyses included diverse conditions, ranging from anxiety disorders, OCD, tics/Tourette’s disorder, and self-harming behaviors. Future studies should aim for an even broader evaluation of the iiPAS in additional diagnoses and symptoms that are currently treated with internet interventions, especially in the behavioral medicine field, where about half of the internet treatment trials are conducted [18]. Moreover, although the scale was designed to be used across the life span, the age range of the participants is limited. Further validation work in adult samples is warranted. In addition, the iiPAS was evaluated in clinical trials in Sweden and, albeit, with projects from different research groups, all were located at the same university and used the same internet platform, limiting the generalizability to other contexts. Future studies should evaluate the iiPAS in other languages as well as in different cultural, health care, and academic environments.

Importantly, the iiPAS was designed to be useful in a wide range of internet-delivered, clinician-supported internet interventions of different therapeutic schools. However, the interventions in this study are all based on a CBT framework. An important next step would be to test the iiPAS in different internet-delivered behavioral interventions. Regarding the reliability analyses, an important aspect that was omitted in this study was the calculation of interrater reliability. Due to the scoring format of the iiPAS, this would require 2 clinicians for each patient, which was not feasible at the current stage of evaluation but would be important to take into account in future studies.

Finally, the sample size in this study was big enough for a global evaluation but too small to conduct detailed separate analyses for each specific condition or age group. A larger sample would allow analysis of the association between patient adherence and clinical outcomes in greater detail with regards to specific patient characteristics such as age, comorbidity, and symptom severity at baseline.

To summarize, the iiPAS demonstrated sound psychometric characteristics in a clinical sample of children and adolescents with a broad range of psychiatric and behavioral problems. The iiPAS appears to be a valuable tool to measure and study different aspects of adherence within the context of clinician-guided, internet-delivered behavioral interventions. Further evaluations within different age groups, types of internet-delivered interventions, and patient populations are warranted.