The advent of eHealth interventions to address psychological concerns and health behaviors has created new opportunities and new challenges. Some eHealth interventions are adaptations of established face-to-face interventions; many are created from scratch as electronically delivered interventions. In either case, they provide the ability to optimize the effectiveness of intervention activities and then deliver these activities consistently to a large number of individuals in need.

Ideally, development of any intervention (whether the intervention is delivered electronically or in-person) begins with a clearly delineated program theory or model of change that is grounded in empirical evidence and clinical experience [1,2]. In such a model, intervention activities target specific mechanisms (psychological or behavioral processes) in order to produce desired modifications in health or behavioral outcomes. The use of a theoretical model to guide development of an eHealth intervention appears to be associated with effectiveness. A recent meta-analysis of 85 studies of eHealth interventions for health behavior change found that interventions that made greater use of theory (ie, linking theoretical constructs to intervention techniques) had larger effect sizes [3].

The concept of content validity originates in the arena of psychological and educational instrument development. Content validity of an assessment instrument is one aspect of construct validity [4,5] and has been defined as “the degree to which elements of an assessment instrument are relevant to and representative of the targeted construct for a particular assessment purpose”, page 238 [5]. In this definition, the relevant elements of an instrument may vary based on the method and purpose of the assessment and include item wording as well as the way in which stimuli are presented, how instructions are given, and which situations are sampled [5]. Content validity is conditional rather than an inherent trait of an assessment instrument; it is assessed with regard to a particular purpose or aim of assessment, and a particular targeted population [5]. Quantitative and qualitative indicators derived from expert review of an instrument’s content validity can be useful in identifying missteps and honing content during the development phase of an assessment instrument [5,6].

Our definition of content validity for eHealth builds upon these well-established attributes of content validity in instrument development. We define the content validity of an eHealth intervention as the extent to which its component intervention activities are relevant to the underlying construct (ie, program theory) and likely to be effective in achieving a particular intervention purpose in a specific intended population. We therefore suggest three core dimensions for expert review: relevance, likely effectiveness, and appropriateness for a specific audience. The first dimension, relevance, is the extent to which an intervention activity is pertinent to its intended intervention target as defined in the program theory or model of change, that is, “Is this arrow aimed in the right direction?” The second dimension, likely effectiveness, is the extent to which evidence, theory, and expert judgment would suggest that this specific activity would successfully modify the intended intervention target, that is, “Is this arrow likely to hit the target?” The third dimension is the extent to which the activity is appropriate for a specific intended audience, which may be defined by age, culture, or other factors.

There are several compelling reasons to attend carefully to content validity in the development of eHealth interventions. When intervention content is developed based on a clearly delineated program theory and model of change, not only is the eHealth intervention more likely to be effective, its use and evaluation also advances understanding of the psychological or behavioral processes in which one is trying to intervene [7,8]. However, after the developer of an eHealth intervention has articulated a model of change/program theory, the next steps are fraught with challenges, including a multitude of choices in the design and delivery of intervention activities. A formal assessment of content validity can be a key checkpoint in the design of actual intervention activities to ensure that these activities and processes match the underlying program theory and change model that they are intended to operationalize. Electronically delivered intervention programs (to an even greater extent than manualized in-person interventions) deliver a set of pre-determined and highly observable activities, thereby facilitating review of specific activities during the development process. Results of content validity assessment can be used to hone an eHealth intervention during its development or to better understand unexpected variations in the performance of an existing intervention. Given the expense of developing eHealth interventions, assessing content validity early in the development process (eg, at the storyboard stage) could be cost-effective by increasing the likelihood that costly further development will lead to an effective intervention.

Undoubtedly, most intervention developers strive to achieve this sort of validity and informally assess the extent to which they have succeeded. However, to our knowledge no systematic process for assessing content validity of eHealth interventions has been proposed. Thus, the aims of this paper are to define content validity as it applies to model-based eHealth intervention development, present a feasible method for assessing content validity in this context, and describe the implementation of this method during the development of an intervention. Based on the definition of content validity for eHealth proposed above, we designed a practical method for assessing the content validity of an eHealth intervention, and applied this methodology during our development of the Coping Coach Web-based intervention for school-age children. Formal review by external experts allowed us to evaluate the extent to which intervention activities matched the model of change and program theory upon which we based our intervention development.

We propose a straightforward, systematic approach to obtaining expert review of the content validity of an eHealth intervention. This approach assumes that the eHealth intervention in question has been created based on an explicit program theory or model of change or that an appropriate theory/model can be applied (even retrospectively) to the existing intervention content. Our approach consists of the following steps:

Based on prior literature regarding content validity in instrument development, we propose both item-level and scale-level content validity indices (I-CVI and S-CVI/AV) as quantitative indicators of acceptable content validity [6]. For eHealth interventions, an “item” is an intervention activity-target pairing, and the “scale” is a set of activity-target pairs. The I-CVI is the proportion (0.0 to 1.0) of expert reviewers who rate an item as 3 or 4 on a 4-point scale. The S-CVI/AV is the average of all I-CVIs for a set of items [6]. Polit et al [6] proposed standards for content validity based on a review of the literature and examination of the quantitative properties of alternate content validity indices. We propose to adopt these standards for eHealth, such that a set of eHealth intervention activities can be said to have excellent content validity if all I-CVIs are at least .78 and the S-CVI/AV is at least .90.

Utilizing the results of content validity assessment to hone and improve an eHealth intervention will always involve both quantitative indicators and the considered judgment of the development team. For example, if a quantitative indicator such as the I-CVI indicates problems with a specific activity-target pair, the next steps for the development team depend on the nature of the problem identified. If an activity is rated as not relevant to its intended target, the development team may consider removing it or undertaking a major revision. On the other hand, if an activity is rated as relevant but not likely to be effective, the development team should consider whether there is a way to alter or enhance the activity to increase its likely effectiveness. Narrative comments provided by expert reviewers as part of the Content Validity Survey can be helpful, and follow-up interviews to elicit additional details about specific concerns may be useful. Developers may also need to take into account whether an activity is rated as relevant/likely effective for some, but not all, of its intended targets.

A total of 15 experts (from the United States, Australia, United Kingdom, Netherlands, and Switzerland) were invited to participate via email; all 15 agreed to participate and provided ratings. All experts were independent, that is, not involved in the development of the intervention. Each expert was an active clinical researcher (12 psychologists and 3 psychiatrists) with at least 5 years of experience in this field and relevant content expertise. Thirteen experts provided ratings within approximately 2 months as requested; 2 experts required additional time due to other commitments but eventually provided ratings. There were very few incomplete ratings (only 2 of 225 ratings for relevance, 3 of 225 ratings for effectiveness, and 5 of 225 ratings for age-appropriateness were missing).

Table 1 shows the I-CVI for each intervention activity/target pairing, based on expert ratings from the Content Validity Survey Tool.

The S-CVI/AV for ratings across all activities was excellent for relevance (.94), although not quite at this standard for likely effectiveness (.82) or age-appropriateness (.85). Examining the quantitative indicators at a more granular level, the I-CVIs for likely effectiveness and age-appropriateness were excellent (≥.78) for nearly all activity-target pairings. However, we identified two activities (encompassing three activity-target pairs) with I-CVIs below .78. Intervention activity 1 (Face-O-Matic Machine activity) and intervention activity 4 (conversation between Coping Coach and General Malaise character), had excellent ratings for relevance but had I-CVIs of .60 to .73 for likely effectiveness or age-appropriateness. Reviewers also provided narrative comments to explain their concerns and/or suggest alternate approaches. Based on these I-CVIs and review comments, these two activities are under review to determine whether they should be retained, removed, or modified in the next iteration of the intervention.

Beyond quantitative ratings, for many intervention activities the reviewers’ narrative comments were helpful in understanding both strengths and potential gaps in this iteration of the Coping Coach intervention. Reviewers commented on likely mechanisms of action, for example, “One of strongest sections, teaches link between thoughts, feelings, and actions well, and good in identifying concrete thoughts” and “Interactive nature of the exercise and the fact that it doesn’t ‘sugar coat’ that there are some positives to avoidance is useful as it makes it realistic for kids”. Reviewers also highlighted ways to extend or improve current intervention activities to better achieve key targets, for example, “Perhaps also discussing what a child’s behavioral reactions may be when sad, angry, worried, etc (for example, crying, stamping feet, churning stomach) may offer them more of a chance to identify their feelings”.

This project demonstrated that assessment of content validity was straightforward and feasible to implement and that results of this assessment can provide useful information for ongoing development and iterations of new eHealth interventions. Expert ratings on the Content Validity Survey Tool demonstrated variability, suggesting that response options were appropriately scaled and anchored to capture useful gradations in expert judgment about the content validity of specific intervention activities.

Especially for components believed to be key to intervention outcomes, assessment of content validity could reduce the number of iterations needed to produce an effective eHealth intervention. The clear articulation of a model of change and program theory, and content validity assessment to ensure that intervention activities match their intended targets, may be especially important in the development of eHealth interventions that are created from scratch, that is, that are not Web-based adaptations of an existing well-established face-to-face intervention [10,18]. However, content validity assessment may also be beneficial to ensure that well-established face-to-face interventions are successfully translated for Web-based/digital delivery. The translation of in-person treatment components (eg, exposure to address anxiety symptoms) is not always straightforward.

Expert ratings of content validity can be an important complement to other sources of information. Depending on the point in the development cycle when content validity is assessed, an intervention development team may need to weigh information regarding suboptimal content validity of specific activities in the context of user feedback, effectiveness evaluations, and other contextual considerations to determine appropriate action. One possibility is immediate refinement or removal of potentially problematic activities; another option is ongoing monitoring of these activities in terms of user engagement or effectiveness. In the case of Coping Coach, expert review affirmed the relevance of all rated activities but identified potential gaps in the likely effectiveness and age-appropriateness of two intervention activities. This information will be utilized in conjunction with user feedback and results of a pilot randomized trial to make decisions about optimizing Coping Coach intervention activities.

Implementation of this method revealed a number of lessons regarding the process and timing of assessing content validity. Regarding process, we learned that thoughtful judgment by the development team is required to define intervention activities at an appropriately granular level, that is, with just enough specificity for meaningful evaluation by expert reviewers. Thoughtful judgment is also required to identify which activity-target pairs merit assessment of content validity. In the case of Coping Coach, we chose to structure the survey with each intervention target paired with the one or two activities that addressed that target most directly. However, some intervention targets are addressed at least indirectly by additional activities and a longer survey form could have asked expert raters to assess all such activity-target pairings. One potential outcome of Steps 1 and 2, or of feedback received during expert review (Step 4), is that intervention developers may realize that they have not adequately specified intervention targets or the intended match between intervention activities and intended targets. If this occurs, it can be seen as an important reminder for the development team, aided by expert consultation if necessary, to revisit and clarify the program theory and model of change that underlie the eHealth intervention. Clarity in this regard is likely to be helpful not only in intervention design, but also in promoting effectiveness of the intervention.

Regarding timing, we sought expert review of content validity at a point in the development process when we had already created a functional online intervention, piloted this intervention with child users, and initiated a pilot randomized trial. The advantage of this timing is that experts saw a fully developed version of the intervention activities and could fully grasp our intended design. There would be different advantages to seeking expert review earlier (with storyboards or functional prototypes) or at multiple points in the development process, namely the ability to iteratively revise an earlier draft of the intervention based on content validity assessment.

There are several limitations of this project that suggest future research directions. First, we suggest the application of quantitative indicators (the I-CVI and S-CVI/AV), and threshold levels for those indicators, which are based in content validity research for the development of psychological measures. While we believe this is a reasonable place to start, additional research is needed to document the range of I-CVI and S-CI/AV results in the development of a variety of types of eHealth intervention activities, and the relationship of these ratings to improved performance of eHealth interventions. Such research would also help to assess the reliability and validity of the proposed Content Validity Survey Tool itself, as this was beyond the scope of the current study. It is important to note that we implemented this content validity approach for an eHealth program with a “tunnel” design, in which all users are required to participate in all activities. Because eHealth interventions vary in the extent to which every user is directed to participate in the same set of activities, assessment of content validity for more complex branching structures, or for interventions that allow free exploration of a set of activities, may require some adaptation of our method. The proposed method for content validity targets three domains: relevance, likely effectiveness, and appropriateness for a specific audience. As the tool becomes more widely used, a need might be recognized for additional domains. Finally, it is essential to remember that content validity as rated by experts is no guarantee of the effectiveness of a set of intervention activities. Researchers and intervention developers should not substitute content validity assessment for rigorous assessment of intervention effectiveness.

Content validity assessment can be a helpful checkpoint in the process of developing or improving an eHealth intervention. Our team created and implemented a straightforward method and Content Validity Survey Tool that provided useful information regarding the match of intervention activities to underlying program theory. This approach could be appropriately utilized at multiple points during the development process to guide ongoing optimization of eHealth interventions.