The meta-analysis was primarily performed to answer the question "How well do coachees adhere to technology-mediated insomnia interventions and diagnostic tools?" and, secondly, to answer the question "How does adherence relate to treatment outcome?" Various databases were queried—Web of Science, Scopus, PubMed, and PsychINFO—on July 8 and 14, 2014, to find studies that investigate insomnia regimes mediated by technology. The used query was: insomnia and Internet-treatment, Internet-delivered, Internet-based, Internet-administered, Internet intervention, computerize, online treatment, Web application, Web-based, virtual, virtual reality, mass media intervention, smartphone, mobile phone, mobile technology, text message, handheld, or PDA (personal digital assistant). In addition, the references from recent meta-analyses, and systematic reviews on self-help and computerized insomnia therapy [16,29,30] were screened for potentially relevant publications. Together, this resulted in 448 unique papers of which the abstracts were read and examined (by the first author, CH) for meeting the following exclusion criteria: no main focus on insomnia, no technology involved, treatment that does not include assignments at home, no experiment, or targeted at children. Studies on children were excluded because children’s sleep problems often differ from those of adults. Besides, children’s bedtimes are partly controlled by the parents. Therefore, interventions targeted at children have other characteristics than interventions for adults and were excluded. A total of 56 papers were read completely and the inclusion of those papers was discussed between the first and second author (CH and JL).

Figure 1 shows the flow diagram for inclusion and exclusion criteria, resulting in 21 papers from which data was retrieved. Due to a lack of reported adherence data in 3 of the papers, only 18 papers were used in the analysis. The papers selected for this meta-analysis can be found in Multimedia Appendix 1.

Of the 18 included studies in this meta-analysis, 12 studies (67%) focused on CBT-I (Table 1 and Table 2). Oosterhuis and Klip [31] and Rybarczyk and colleagues [32] did include most of the CBT-I exercises in their intervention. Out of the 18 studies, 2 (11%) focused on sleep tracking by using an active sleep sampling device. The active sleep sampling device used by Riley and colleagues [33] mainly supported sleep restriction and stimulus control. The other standard CBT-I exercises were explained in an additional manual. Lawson and colleagues [34] used an active sleep sampling device inspired by Riley’s device. They developed an active sleep sampling mobile phone app which focused on sleep tracking, but did not include the other CBT-I components. Lipschitz and colleagues [35] also developed a mobile phone app, offering sleep-focused, mind-body bridging exercises. The most important assumption of mind-body bridging for sleep is that the mind needs to be rested to sleep well. Haimov and Shatil [36] studied whether providing cognitive training, such as a memory game, affects sleep.

In order to establish a purposive sample of users across sleep products, various sleep products were categorized. Based on their background knowledge and a media scan, the authors generated a list of 54 technologies over the course of a few months. This composed list was supplemented with apps because the goal of the Sleepcare project is to design a virtual sleep coach on a mobile phone. The first 25 Android apps and 25 iPhone apps found in Google Play and the iTunes store with the search word "sleep" on November 19, 2012, were added to the product list. A total of 7 apps were unrelated to sleep—3 games, 2 hypnosis apps, 1 unlock, 1 music timer—and were therefore discarded, resulting in a list of 97 sleep products. The categorization made in this paper aims to be simple and objective. Sleep products were categorized based on their goal and the medium used. Figure 2 shows the distribution of the products across the two dimensions: goal and medium. The size of the bubbles shows how many products belong to the intersections of the categories.

After identifying the categories of existing sleep products, the next step was to learn more about the users’ usage and adherence to the sleep products. Interviews were conducted with people who used a sleep product in each of the largest product-medium combinations (eg, apps that help people fall asleep).

People registered as participants at the Sleepcare project website [48] were invited to participate in the interviews if they had ever used a technology-mediated sleep product. In addition, two sleep therapists were asked to invite people who used sleep coaching products, as none of the respondents to the call used a sleep coaching product. A total of 15 Dutch persons agreed to be interviewed—6 (40%) females and 9 (60%) males—their ages ranging from 22 to 65 years (mean 37.5, SD 14.8). The mean Pittsburgh Sleep Quality Index (PSQI) [49] score was 8.0 (SD 4.0), with 12 out of the 15 (80%) interviewees having a score above 5, which is the threshold for poor sleep quality classification.

Besides adherence, the interviews covered other topics to gain insight regarding users’ experiences with sleep products. Therefore, the semistructured interviews included both adherence-related questions and questions regarding the factors of the Unified Theory of Acceptance and Use of Technology (UTAUT) model [50]. The described results of the interviews in this paper, however, will only include adherence-related topics. The interviews were conducted in person, by Skype, or by telephone by the first author. The audio of the interviews was recorded. The study was approved by the Human Research Ethics Committee of Delft University of Technology.

The first author (CH) performed the data analysis following the phases of thematic analysis as described by Braun and Clarke [51]. The first author familiarized herself with the data (phase 1) by conducting and transcribing the interviews, and reading the transcripts. While reading, the initial codes were generated bottom-up (phase 2). The first author coded the transcriptions and iteratively generated hierarchical codes and themes (phase 3). Short summaries of the codes and themes related to adherence were written down. The first and last author (CH and WB, respectively) discussed these summaries (phase 4) to form three final adherence-related categories (phase 5). In addition, an independent researcher applied the coding scheme to one of the interviews in order to minimize the threats to confirmability (known as objectivity in quantitative research). The independent coder confirmed the applicability and usefulness of the codes.

The envisioned coach would use different adherence-enhancing strategies during the entire coaching process. For example, different roles (eg, motivator and educator) could be played by different virtual characters to increase the effect of the to-be-developed sleep coach (ie, split-persona effect) [52]. Around 25 strategies were allocated to the coach ranging from strategies involving others (eg, peers or family members), helping with planning (eg, setting goals and making commitments), and gaming strategies (eg, earning points and taking a quiz). These adherence-enhancing strategies were scripted explicitly in the scenarios in order to discuss them in the focus groups.

A total of 12 scenarios and 72 claims (see Multimedia Appendices 2 and 3) were written to evaluate the adherence-enhancing strategies. Scenarios consisted of stories about people and their activities, goals, and motivations regarding a system [53]. Claims stated important design decisions (eg, about the adherence-enhancing strategies) that needed to be evaluated in the focus group. Furthermore, three fictitious people varying in age, gender, family situation, and readiness-to-change were created to act in the scenarios (see Multimedia Appendix 4 for these personas).

The scenarios and claims were discussed in six focus groups to evaluate the adherence-enhancing strategies. Two groups consisted of potential users, two groups consisted of coaches, and a further two groups consisted of sleep experts. Demographics of the Dutch participants can be found in Table 3. Each session lasted 2 hours and included a general introduction, an introduction round of the participants, and approximately four animated videos that represented the different scenarios. After watching one video, the participants were asked to individually rate their agreement with the claims on a 7-point Likert scale. Subsequently, participants were asked in turn to react to the claims and discuss their ideas. The sessions were videotaped for later analysis. The study was approved by the Human Research Ethics Committee of Delft University of Technology.

The analysis was an iterative process of developing codes and themes in line with thematic analysis [51]. For that, the videotapes of the sessions were transcribed and summarized by the first author (CH). During that recapitulation, several codes emerged and an initial coding scheme of 12 codes was created. The first author coded the summaries according to this scheme. Additionally, a second coder, independent of the project, coded a sample of the summaries—48 of the 86 claims (56%). The second coder suggested eight additional codes. The two coders came together to discuss the coding scheme and agreed on a new scheme of 15 codes. The coding was improved (with this new scheme) by both coders, and within the sample a Cohen’s kappa of .80 was reached. Next, the first author wrote short resumes per theme, making use of quotes.

The three main categories related to adherence are usage, effectiveness, and adherence (see Textbox 1).

Two initial reasons for using a product emerged from the interviews. First, all interviewees used the product to overcome some of their sleeping troubles. Interviewees wanted to wake up better, initiate or maintain sleep, and/or increase insight into their sleep. Second, some interviewees used a product because they thought the product in itself was interesting. Above all, this holds for the sleep-tracking apps. Most products—alarms, automatic sleep trackers, and relaxation support—were used in the evening before going to sleep. Most of those products, however, were not used on the weekend. Sleep coach usage varied, depending on the kind of assignments included in the product (eg, diary, relaxation, sleep hygiene exercise, bedtime scheduling).

Participants' quotes regarding reasons to start using sleep products were as follows (translated from Dutch):

Participants' quotes regarding usage of sleep products were as follows (translated from Dutch):

One of the initial arguments for using a product was to overcome some kind of sleeping problem. However, the online sleep therapies were not perceived as having an effect on the interviewees’ sleep problems. Additionally, interviewees mentioned that it was hard to determine if the therapy improved sleep in the long term because they tried several things. Nevertheless, most interviewees took some advice that worked for them and continued applying it. Furthermore, sleep tracking apps as well as online sleep coaches provided the interviewees with more insight into their sleep and habits. Products that wake interviewees up (ie, smart alarm apps and wake-up lights) were assessed ambiguously. Both types of products did what was expected of them, namely wake the interviewee up. However, the effect of waking up better with the product was doubted. Moreover, smart alarms did not seem to fit into interviewees’ daily lives (see quote below from interview #4, sleep tracking app).

Participants' quotes regarding the effectiveness of sleep products were as follows (translated from Dutch):

In general, interviewees perceived their own usage as sufficient. Interviewees especially perceived their own personal attitude, beliefs, and willpower as important for adherence. These personal characteristics were regarded as particularly important for adherence to therapy-related products. The usage of consumer products (eg, an alarm clock) was continued, because the interviewees needed the functionality. The main arguments for not using a consumer product were (1) no perceived need for the product, (2) a perceived lack of effectiveness, and (3) the interviewee forgot to use the product.

Participants' quotes regarding the satisfaction about adherence of sleep products were as follows (translated from Dutch):

Participants' quotes regarding the effect of personal attitude, beliefs, and willpower on adherence of sleep products were as follows (translated from Dutch):

Participants' quotes regarding reasons for using and not using sleep products were as follows (translated from Dutch):

The most obvious emerging themes in the focus groups were users in control and doing it for your own sake. In general, participants believed in the personal strengths and willpower of users to adhere to the proposed sleep coach. Furthermore, the adherence-enhancing strategies and motivation were discussed. See Textbox 2 for an overview of the results.

Potential users, coaches, and sleep experts agreed that the users should be in control. Different arguments were given. The coaches and sleep experts mainly argued that giving the user more control increases commitment and motivation. The potential users argued that they use the sleep coach for their own sake, so they want to be in control themselves. Another argument was that not being in control could lead to irritation. Aspects that participants believed the users should be able to control were the following: reminders, amount of information given by the app, scheduling exercises, decisions about motivation level, sharing therapy progress, sharing the outcome of questionnaires, and parameters shown in sleep diary overview.

The other interesting theme was doing it for your own sake. In one scenario, there was an example exercise which entails making a list of people who can help you. In general, this exercise met resistance by the potential users. The idea that you have to solve your problems yourself was dominant. Users would feel ashamed to ask for support, and they believed the virtual sleep coach should help them. On the other hand, the coaches stated that thinking about social resources, such as family and peers, could really help people. The coaches mentioned that coachees usually consult a coach exactly because they try to solve their problems themselves, instead of asking their social resources for help. One potential user shared that only informing other people about her sleep problems and therapy already helped her a lot, even without asking for support. Nonetheless, the general mind-set was that people use such a coach for their own sake, and that they are and should be able to take responsibility for their own adherence.

The claims underlying the envisioned usage scenarios stated that users should be motivated before they start sleep treatment, otherwise the probability of dropping out would be too high. The focus groups with the sleep experts manifested three different ideas about motivation. Some of the sleep experts argued that people will be motivated at least a little bit when they have downloaded the app, since that requires some effort. On the other hand, it was also argued that someone could show interest in the sleep coach, but he or she would not necessarily be motivated to use the sleep coach. Third, it was argued that motivation could arise during different phases of a therapy; for example, after someone performs an exercise and experiences its effects. In that situation, users would not need to be highly motivated at the beginning of the therapy.

The interviews and focus groups both revealed that people strongly believe willpower is an effective adherence strategy. Participants believed that their personal attitudes, beliefs, and motivation would ensure that they stick to their intentions of using a product. This result should be interpreted with caution because of three phenomena. First of all, sleep deprivation increases ego depletion [56]. In other words, when people are tired their willpower decreases and it will become more difficult to adhere to anything, including a virtual sleep coach. Second, the interviewees attributed their adherence to their own commitment and attitude, while nonadherence was attributed to malfunctioning of the product. This result should also be interpreted with caution because this phenomenon is in accordance with the self-serving bias. The self-serving bias states that successes are attributed to internal factors, while failure is attributed to external factors [57]. Therefore, the "good" adherence rates in the interviews were attributed to the interviewees' own willpower. Third, the participants in the focus groups were quite optimistic about their anticipated future adherence. Being optimistic about oneself and the future is one of the most robust biases (optimism bias) in psychology [58,59]. Several explanations for this unrealistic optimism has been offered, for example, ignoring everything that could go wrong [58], putting too much weight on current intentions [60], or having too much faith in willpower for future events [61]. These three phenomena provide reasons for treating participants’ optimism toward adherence with caution.

Apart from relying on willpower for adherence, aversion to adherence-enhancing strategies emerged during the focus groups. Therefore, when designers implement adherence-enhancing strategies they should not assume that users would initially agree with the usefulness of these strategies.

Various design principles for a virtual sleep coach can be adopted from the interviews and focus groups. The first design principle covers functionality. During the first usage phase, the sleep coach should immediately tickle users’ interest, for example, by providing automatic sleep tracking. In the interviews, it appeared that interest made coachees start using products. Next, the sleep coach can provide an already-needed functionality (eg, an alarm clock). According to the interviews, a needed functionality ensures that users keep using a product. Lastly, reminders need to be a part of the sleep coach. Reminders make sure that users do not simply forget to adhere to the coach. Both the participants in the interviews and focus groups indicated that sometimes they just forget to use a product. Participants in the focus groups showed a positive attitude toward reminders as long as the users were in control over the reminders. Therefore, including reminders in a sleep coach would be a good first step in future research to increase adherence.

A second design principle could be to withhold adherence support at the start of the intervention (ie, to postpone possible help by a virtual sleep coach). In this way, the coachees are acknowledged and respected as serious, motivated, and autonomous adults. Coachees can prove that they adhere to the assignments of the sleep coach; however, the virtual coach can detect when coachees fail to do their assignments, and then offer support. This support can take different forms (reminders, compliments, awarding points, etc) and can be varied over time based on the needs of the coachee.

A third design principle that can be applied is explaining why willpower does not guarantee success. After such an explanation, the understanding of the added value and acceptance of adherence-enhancing strategies might increase. On top of that, users could be given the control over the employment of adherence-enhancing strategies.

In the authors’ opinion, the most important overall design principle is balance. Coachees should not feel overwhelmed with adherence-enhancing strategies, but appreciate some occasional support. Personalization of the virtual sleep coach can ensure that the perfect balance is reached for each and every user. For example, some users might need and appreciate reminders for filling out a sleep diary every day, while other users are more likely to forget to do their relaxation exercises.

Lastly, we want to stress that studies should measure and report treatment adherence, and make a distinction between experimental compliance and treatment adherence. It is important that future studies measure and report adherence rates, since it is only by the adherence measure that it can be established whether the treatment actually induces the observed outcome. The frequently made statement that adherence is important for the outcome of a treatment [17-19] seems to be supported by the findings of the meta-regression between effectiveness and treatment adherence. As a correlational analysis does not provide insight into the direction of a causal relationship, it remains unclear how effectiveness and adherence influence each other. Nevertheless, if coachees do not follow the treatment protocol (ie, adherence rates close to zero), the outcome could be attributed to other things outside the intervention [21], for example, to the waiting-list effect. Furthermore, it is important to make a distinction between experimental compliance and treatment adherence, since these seem to be two distinct constructs as the meta-analysis found no correlations. An earlier meta-analysis about the effectiveness of CCBT-I found a rather good "adherence rate" of 78% [16]. However, this rate would be considered as experimental compliance according to the definition used in this paper. Similar experimental compliance rates—79%, 72%, 70%, and 57%—were found by the meta-analysis, although treatment adherence was significantly lower. The average self-reported treatment adherence was 42%, whereas the logged treatment adherence was 64%. Although no significant difference between these two measures was found, it is important to consider how adherence is defined and measured. A study [62] that compared a paper diary with an electronic diary found a tremendous difference between self-reported adherence (90.5%) and logged adherence (10.9%) for a paper diary. Lastly, the question remains whether adherence in experimental settings resembles adherence in nonexperimental real-life settings. It could be that adherence rates in experiments are higher than in real-life situations. One possible explanation is the sunk-cost fallacy [63]. To illustrate, experiments demand more from participants regarding (pre-) measurements and participants might therefore be more committed to the intervention. When starting a treatment, they have already invested more time (ie, the sunk cost) compared to patients in nonexperimental settings, and are therefore less likely to drop out.

In order to review the quality of our research, it is helpful to know what we did to take care of the credibility, transferability, dependability, and confirmability of our studies [64]. Firstly, threats to all four concepts were minimalized by utilizing three different research methods—meta-analysis, interviews, and focus groups. Furthermore, the credibility of our findings is also consolidated by data source triangulation—literature, current users, potential users, coaches, and sleep experts. Additionally, honesty from our informants was reinforced by stating there are no right or wrong answers, and by allowing them the possibility to withdraw at any moment. We also had regular debriefing sessions between the executors and supervisors in order to strengthen credibility. The level of transferability to other application fields can only be judged by the readers, since they have the knowledge of these other domains [65]. Furthermore, future work can be done to replicate these findings in other fields. Transferability and dependability assessments are supported by descriptions of the research methods and Multimedia Appendices. Lastly, confirmability was addressed by audit trials and the second coders.