Effectiveness refers to how well a health or psychological treatment works in a real-world setting [19]. Methods used for clinical trials of pharmacological treatments can be applied to complex nonpharmacological interventions such as eHealth initiatives [20]. Randomized clinical trials (RCTs) have found that Web-based interventions are effective in treating a range of psychological issues. For example, individually tailored Internet-delivered cognitive behavioral therapy (CBT) has been found to be an effective and cost-effective means of treating patients with anxiety disorders [9]. A recent meta-analysis of RCTs investigating the effectiveness or efficacy of Web-based psychological interventions for depression found positive results across diverse settings and populations [21]. However, as Richards and Richardson [21] note, effectiveness is only one part, albeit an important one, of an intervention that should be evaluated. For example, effectiveness research does not necessarily investigate the influence of therapist factors.

In evaluation research of face-to-face psychosocial interventions, findings have shown that an effective therapeutic relationship in and of itself may be enough to provide successful outcomes for clients [22]. Moreover, this correlation between ratings of the therapeutic relationship and outcome seems unaffected by other variables such as outcome measure or type of intervention employed [23]. These variables are client derived, based on their subjective experiences, and can be adjusted for and evaluated throughout the treatment program or therapy session. As a result, the interaction between the client and clinician is fine-tuned and dynamic, and the clinician is able to constantly monitor the therapeutic relationship and make changes accordingly. This is possible in a dynamic person-to-person interaction but it is not possible in a one-way interaction between a person and a computer.

In the person-computer interaction, the psychological experience of the person using the therapeutic intervention is lost and cannot be responded or adjusted to (although there are developments in this area [24]). This raises three key concerns. First is the potential loss of the therapeutic relationship. Second, the psychological experience of the person may be important to determining usage and adherence to an intervention with some reports of dropout rates exceeding 98% from open-access websites dealing with psychosocial issues [25]. Third, it may be important in understanding issues of psychological safety and well-being for “if information is too complex to understand, especially under periods of duress or high cognitive load” then the intervention may not be as effective [6]. Therefore, the in situ psychological experience of the user may be an important variable to be considered; however, it is not clear how it has been or should be evaluated.

In traditional face-to-face interventions, evaluation of user experience is relatively straightforward. For starters, the client can provide instant and direct feedback to the therapist during a session. In addition, there are a number of psychometrically reliable and valid standardized tools such as the Helping Alliance Questionnaire [26] or the Outcome Rating Scale and Session Rating Scale [27], which directly address user experience. In e-therapy or e-counseling, when there is still a person-to-person relationship via Skype or email, the measurement of the therapeutic process is similar as only the delivery differs [12]. However, when the intervention is provided by a website and is driven by computer-programmed algorithms and automated responses it is much more complex. A cross-disciplinary approach is needed that links the psychosocial approach with user-centered design from computer science [6]. One of the most common ways of evaluating the relationship the user has with a website or Web-based intervention is through usability testing.

The literature on human-computer interaction yields a number of different methods of evaluating a website, be it a psychosocial intervention or an web-based retail store. One of the most widely used and recognized means of addressing this is usability evaluation [28]. Usability testing is a means of evaluating the design and functionality elements of a website as they apply to the user. However, the end user is not always involved and, if so, any information capturing the user experience is designed to enhance the performance of the intervention in terms of content, design, and navigability of the website, rather than an investigation of the relationship between the user and computer.

Usability inquiry on the other hand is designed to gather broader and more subjective information from the user with regard to the website or intervention. Zhang [29] defined usability inquiry quite broadly as gaining “information about users’ experience with the system by talking to them, observing them using the system in real work (not for the purpose of usability testing), or letting them answer questions (verbally or in written form)” [29]. Therefore, although usability testing provides output from the person using the system, the focus is on functionality of the website. In usability inquiry, there is a subtle shift to understanding the users’ goals, context, profile, feelings, and thoughts during the process of interaction. The output extends beyond issues of website design and functionality to broader concerns of purpose and context [28].

The methods of ethnographic interviewing, contextual inquiry, cognitive interviewing, and situated co-inquiry lend themselves, somewhat naturally, to an exploration of how users may experience a Web-based psychological intervention. They all place the user as the most important piece of the process. Thus, it fits the paradigm of interpretative and participatory research with elements of phenomenological inquiry to gain an understanding of the users’ experience in a natural setting. In some ways, this would come close to understanding the nuance of the relationship between a user and an intervention in a similar way that an evaluation of the therapeutic relationship captures a client’s perception and experience of traditional therapy. However, it is unclear to what extent usability inquiry, or any other methods, have been used to address the psychological experiences of those using Web-based psychosocial interventions. Thus, a systematic literature review was undertaken to assess what methods, if any, have been applied to understanding users’ in situ psychological experience with Web-based interventions.

There were 4 criteria components for selecting studies. First, the health issues of interest were those with a psychosocial component for which a person may have been reasonably expected to see a clinician if the Web-based intervention was not available; for example, helping people with common mental health concerns such as stress, anxiety, and depression. Also included were psychosocial or behavior change intervention programs associated with physical ailments, such as rehabilitation and psychological well-being following surgery or living with human immunodeficiency virus infection. The psychosocial element was required to underscore the importance of the user’s psychological experience with the intervention. Thus, interventions with a purely physical component such as smoking cessation were excluded.

Second, the type of interventions included were Web-based interventions with a primarily self-help basis that required users to work their way through a series of steps. Excluded were e-counseling and e-therapy programs where the treatment was based on person-to-person interaction rather than a person-to-computer interaction. They were also excluded if they were solely psychoeducation or health promotion websites that did not require an interactive, self-guided program. Online tools (such as “Facebook”) that may be used as part of social skills therapy but not designed for the purpose of treatment or psychosocial intervention were excluded.

Third, the method of evaluation had to involve end users’ assessment of their experience, rather than expert assessments. Finally, the focus of the evaluation must be the end users’ psychological experience interacting with the intervention. Studies that looked at the overall efficacy or treatment outcome of a Web-based intervention were excluded. Usability studies were excluded if they focused solely on functional Web-development issues (content, layout, design), rather than on the psychological experience of the user. For example, “think aloud” (also called talk aloud or cognitive interviewing) studies that did not elicit psychological insight or experience from the user but discussed use of menu options, friendliness, and ease of use were not included. Basically, the study had to be user focused, rather than intervention or computer focused. Expert usability and heuristic studies were also not of value as they did not address the actual experience of the intended user of the intervention. If an evaluation of the users’ psychological experience formed part of a broader study on the effectiveness study or feasibility study, it was included. As a result of a scoping exercise, we expanded the search protocol to include end users’ psychological experience per se, whether in situ or postintervention.

Keywords applied in the search were determined by an initial review of the literature and modified by feedback from psychologists, computer science usability experts, and a librarian. Sets of terms were proximally searched including experience (“experience,” “evaluation,” “usability”), Internet (“internet,” “web*,” “online,” “eHealth”), health (“mental health,” “behavio*,” “psycho*”), and interventions (“intervention,” “treatment,” “therapy”). To ensure coverage of the health, psychological, and computing components of the topic the following databases were searched: CINAHL (EBSCO), MEDLINE (EBSCO), and Psychology and Behavioral Sciences Collection (EBSCO), Computers and Applied Sciences Complete (EBSCO), ABI/INFORM Global, and IEEE Xplore (see Multimedia Appendix 1). The search was restricted to studies from 2004 to October 2015 because of the relative newness of the field of Web-based interventions. A title scan across non–peer-reviewed journals elicited no studies of interest in the initial scoping, so only scholarly journals were included in the search. A total of 2 studies came from a review of the references of a meta-synthesis of user experience of online computerized therapy for depression and anxiety [32].

The search protocol resulted in 62 records that were screened by 2 reviewers (LM and LR), with disagreements resolved by the research team (see Figure 1). After final full-text review, 21 studies were identified as meeting the 4 inclusion criteria. Included and excluded studies are listed in Multimedia Appendices 2 and 3, respectively.

The interventions were Web-based self-help interventions consisting of a number of modules (ranging from 3 to 12), most expected to be used on a weekly basis. Although all of the studies were predominantly self-help programs, some were augmented by other elements such as an online diary [37,38,41], social or peer group support in the form of discussion or chat groups [38,39,42,44,46], and some degree of therapist or tailored guidance to support the self-help component [34,38-40,43,53]. A total of 9 studies focused on an intervention in the pilot or development phase [33,34,36,39,43,46,49,52,53], whereas 13 studies focused on existing treatment programs such as “MoodGYM” or “Colour Your Life” [35,37,38,40-42,44,45,47,48,50-52]. All of the interventions contained interactive or homework components and some reported including multimedia such as audio or video in content delivery.

Of the 21 studies, most studies focused on the treatment of psychological issues. A total of 10 studies evaluated interventions designed to treat or prevent the development of depression using principles of CBT as the modality of treatment [34,36,37,40,41,44,45,47,51,52]. One intervention used CBT to assist adolescents with stress, anxiety, and depression [35]. Of the studies, 2 focused on delivering mainstream treatment for eating disorders online [43,48] and 1 on the treatment of first psychotic episodes through the use of CBT, support, and psychoeducation [46]. One intervention used motivational interview principles to support veterans with war exacerbated psychiatric issues apply for jobs and benefits [48]. Another aimed to reduce child behavior problems and parenting stress for young children with traumatic brain injury (TBI) via an online self-guided program and live coaching [53]. A total of 4 studies focused on interventions delivering psychological well-being and support to those with chronic or acute physical issues including irritable bowel syndrome [50], cancer [39,42], and angina [38]. Another study used a positive psychology approach to enhance mood in general populations [33].

Of the 21 studies, 13 were published from 2012 onward (see Table 1). A total of 10 studies were explicitly focused on the evaluation of the patient or users’ psychological experience of the intervention [34,35,38,40-42,44,47,48,50]. Of the remaining 10 studies, 8 gathered user experience as part of the development and pilot testing of the intervention [36,39,43,45,46,49,52,53]. Another study collected information on user experience to assess the acceptability and use of online CBT for depression [37], and another looked at the efficacy of using an online CBT intervention for depression with patients experiencing depression along with their TBI [51].

The rationale for evaluating user experience varied across the studies. Of these, 5 studies were seeking information on the users’ psychological experience of using the website to inform improvements [36,39,45,46,49]. A total of 6 studies were interested in finding out how to increase use of online interventions by understanding barriers to change [35] and acceptability in particular populations [42-44,51,52].

The process of gathering user experience information also varied. A total of 13 studies relied on only one form of evaluation method. Of these, 10 studies used only an interview at the end of the treatment program (with a range of 1 week to 2 years posttreatment, where specified) to ask users about their experience with the intervention. Of the studies, 5 used face-to-face interviews [34,38,41,46,47] and 5 employed phone interviews [35,42,48,50,51]. The interviews were conducted by the researcher or research assistants (when specified) and their duration ranged from 19 minutes [35] to 111 minutes [34]. Of the studies, 3 employed only questionnaires to investigate user experience. Of the 3 studies, 2 deployed it at the end of each treatment session [33,36] and the other [37] at the beginning of treatment and then 3 months posttreatment.

The remaining 8 studies relied on a mixed methods design to investigate user experience. Of these, 4 studies employed a posttreatment semistructured interview, 1 study [39] complimenting it with a posttreatment questionnaire and another study with a posttreatment satisfaction survey [53]; Gega and colleagues [40] with postsession questionnaires and Hind and colleagues [44] with a postsession evaluation form. One study used weekly evaluation questions and a postintervention questionnaire [43]. A further study used a during and after session diary to capture user feedback combined with a posttreatment questionnaire [52] and another used module and a postintervention evaluations along with content analysis of the site’s discussion forums [45]. The final study used a think aloud usability process during the session followed by an end-of-treatment questionnaire [49]. Note that this was the only study employing a usability testing method that included a user experience focus, explicitly asking for feelings during the session, and a questionnaire to elicit user experience with the therapeutic alliance during the Web-based intervention.

Tools used to gather user experience information ranged from customized to off-the-shelf. Of the 14 studies that used some form of interview with users, 12 were semistructured with the topic guide designed by the research team. Of these, 2 studies provided a theoretical basis for the design of the interview guide. Lillevoll and colleagues [47] followed a phenomenological-hermeneutic approach to understand the lived experience of the user of the intervention in the natural setting, and Bradley and colleagues [35] followed a Theory of Planned Behavior. A further 2 studies [34,40] based the interview guide on the Client Change Interview (CCI), which was designed as an empathic exploration of aspects of a client’s experience with traditional face-to-face counseling and therapy.

Questionnaires used to evaluate user experience were either researcher designed or existing tools. Of the studies, 7 used self-customized questionnaire or evaluation forms [33,37,43-45,52,53]. Others relied on questionnaires designed to gain feedback about the therapeutic process, sessions, expectations, and outcomes (Table 2).

As this study is focused on the methods of evaluating user experience as opposed to user experience per se, the results of the studies are not presented in detail. In summary, quantitative analysis was carried out on the statistical data gathered in the questionnaires and descriptive summaries reported where appropriate. The qualitative user experience information in 19 of the studies was analyzed using content analysis to elicit emerging themes. Of these, 2 studies followed elements of grounded theory methodology to develop theories of user experience with Web-based interventions [41,52]. One study took a phenomenological-hermeneutic approach [47].

Overall, these studies reported thematic categories to describe the users’ experiences within the frame of the study purpose. Thus, the themes that evolved were varied and included user experience in terms of the therapeutic process [40,41], individual and social aspects of the intervention [33,40-42,47], as well as Web-based characteristics of the intervention [40-42,48-50]. Process issues included motivation and user profile [34], privacy and help-seeking [35], overall experience [37,46], barriers [50,52], expectations [37], personalization, solitude, and social, individual, and contextual features [40,41]. For example, in one of the grounded theory studies, Gerhards and colleagues [41] identified themes of the user experience in terms of the computer, individual, social, research, and environmental contexts.