This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Understanding the preferences of potential users of digital health products is beneficial for digital health policy and planning. Stated preference methods could help elicit individuals’ preferences in the absence of observational data. A discrete choice experiment (DCE) is a commonly used stated preference method—a quantitative methodology that argues that individuals make trade-offs when engaging in a decision by choosing an alternative of a product or a service that offers the greatest utility, or benefit. This methodology is widely used in health economics in situations in which revealed preferences are difficult to collect but is much less used in the field of digital health. This paper outlines the stages involved in developing a DCE. As a case study, it uses the application of a DCE to reveal preferences in targeting the uptake of smoking cessation apps. It describes the establishment of attributes, the construction of choice tasks of 2 or more alternatives, and the development of the experimental design. This tutorial offers a guide for researchers with no prior knowledge of this research technique.
Understanding how the public values different aspects of digital health tools, such as smoking cessation or physical activity apps, can help providers of the tools to identify functionality that is important to users, which may improve uptake (ie, selection, download, and installation of apps) [1]. This is important because uptake of digital tools is generally low. More information regarding the preferences of users when selecting a digital health tool, for example via an app store, may allow providers to present their products in such a way that may increase their uptake. However, pragmatic challenges, such as examining how each potentially modifiable aspect of a digital health product (eg, presentation, design, and features that it offers) or intervention design will impact preference or the choice of uptake, often mean this is not feasible or practical [2]. Therefore, increasing attention is being paid toward stated preference methods to understand preferences when designing digital health products and services, with examples including COVID-tracing apps [3,4], sun protection apps to prevent skin cancer [5], and the uptake of health apps in general [6].
Stated preference methods are survey-based methods aiming to elicit individuals’ preferences toward a specific behavior, particularly those that are not well understood. The most widely used type of stated preference method is the discrete choice experiment (DCE) [7]. According to Spinks et al [8], Louviere and Hensher (1982) and Louviere and Woodworth (1983) originally developed DCEs to study the marketing and economics of transport, and the fields of psychology and economics have profoundly influenced the DCE methodology since it was developed. In recent years, DCEs have been increasingly used in health and health care settings [9,10], as well as in addiction research [11] and digital health [4-6]. The increasing number of DCEs in digital health highlights their potential, although they are currently underused.
Discrete choice differentiates from other stated preference methods in the way that responses are elicited [12]. The DCE uses a survey-based experimental design, where participants are presented with a series of hypothetical scenarios. In these scenarios, participants are shown situations, known as choice tasks. Attempting to mimic real-world decision making, in each choice task, participants then have to choose a product or a service from two or more options, known as alternatives [13]. Each alternative consists of a set of characteristics, known as attributes, with at least two types, known as attribute levels [13]. Participants are asked to choose a preferred alternative in each choice task, which allows researchers to quantify the relative strength of preferences for improvements in certain attributes [8,14].
The outputs from statistical models developed using DCE data can be beneficial for estimating uptake of new products or services, including digital health tools, where observational data are not available or are difficult to obtain otherwise [15,16]. Lack of observational data often implies a requirement to seek scientific views and comments from experts in order to generate predictions of a target behavior [17]. However, DCEs can provide an empirical alternative to expert opinions, while accounting for possible interactions between attributes (eg, design of a product and brand name), which are otherwise often ignored [18].
In our research, we wanted to understand how to present health apps on curated health app portals to increase their uptake. This paper describes the development of a DCE in digital health that aims to elicit potential user preferences on smoking cessation app uptake. It explains how the attributes and their levels are selected and describes the construction of choice tasks and the experimental design. The study protocol of the research this paper is based on is registered on the Open Science Framework [19].
Development of a DCE
The development of a DCE should follow published recommendations, including the checklist for good research practices [9], guides on the development of a DCE [13,20], recommendations on how to construct the experimental design [7,20-23], and which statistical methods can be used [24].
Establishing Attributes
An important step in designing a DCE is the identification of the relevant attributes for the subject matter. Attributes in a DCE can be quantitative, such as cost, or qualitative, such as the design of a product [25]. The identification of attributes is typically based on primary and secondary data collection to ensure that the DCE is tailored to the study setting [13]. It should ideally commence with a literature review that will inform qualitative research to identify relevant attributes [26]. Although there is no set limit on the number of attributes that can be included in a DCE, to ensure that the cognitive load of the participants is manageable, it should be less than 10 [13], with a general expectation to include 5-7 attributes [27].
Our DCE was based on a comprehensive systematic review investigating factors influencing the uptake and engagement with health and well-being smartphone apps [28] and a qualitative research component that consisted of a think-aloud and interview study to examine further the previously identified factors or attributes [29]. The importance of qualitative research lies in ensuring inclusion of attributes that are relevant to most participants [25]. Of the 14 factors initially identified as being relevant for the uptake of health and well-being apps, 5 were retained and included in the DCE: the monthly price of the app, who developed the app, the star ratings of the app, the description of the app, and images shown. These factors were chosen due to their perceived importance during our previous qualitative research and for pragmatic reasons, including how easily measurable and presentable they were within the DCE.
An important step in designing a DCE is in ensuring the content validity of the instrument: the identification of relevant attributes for the subject matter. Following administration of the survey, methods are available for the measurement and assessment of the content validity of the instrument, although their use is not widely reported [30].
Establishing Attribute Levels
The next step is to establish attribute levels. The level of an attribute must also be of a range that ensures a trade-off between attributes. A trade-off is defined as an exchange in which a participant gives up some amount of one attribute to gain more of another. It has been suggested that increasing the number of levels for an attribute increases the relative importance of that attribute [31] and that imbalance in the numbers of levels across attributes raises the importance of the attributes with higher levels [32]. Yang et al [32] suggested that a balance exists between simpler designs with lower numbers of levels, which reduce the respondent burden (and consequently measurement error) and are useful for identifying attribute rankings, and more complex designs with higher levels (and higher statistical precision) and is more sensitive to identifying trade-offs between attributes. Based on this, and the commonly adopted practices in the research field, we aimed to include at least three levels for each attribute.
If a range is not suitable, participants might consider the differences between levels unimportant [25]. For example, the difference between the star ratings of 4.8 and 4.7 for a smoking cessation app is not as relevant as the difference between 4.8 and 4. In our research, to refine attribute levels, a survey was conducted with 34 participants. In the survey, the levels of two attributes we were unsure of (the monthly price of the app and the ratings) were carefully considered in order to specify at a sufficiently wide range so that the difference between the levels would likely make a difference in response. When a range is not wide enough, there is a risk that participants could ignore the attributes because they judge the difference between levels to be insignificant [20]. See Figure 1 for the final list of attributes and levels included in our DCE.
Attributes and attribute levels in our DCE. DCE: discrete choice experiment.
Choice Tasks
Once the attributes and their levels are identified, the decision to develop full- or partial-profile tasks with or without an opt-out option needs to be made. A full profile refers to the display of all five attributes in both alternatives in each choice set. A partial profile DCE will not present certain attributes for certain alternatives. For example, if a DCE is used to investigate the trade-off between a higher number of attributes (eg, a total of nine attributes), it could be beneficial to limit the number of attributes shown at one time (eg, five attributes) to limit participant cognitive load. Five attributes are generally considered low enough to complete a full-profile choice task, which consequently maximizes the information about trade-offs [33]. Hence, in our research, we applied a full-profile DCE.
A neutral option (“Neither of these 2”), known as an opt-out alternative, was included, in addition to selecting alternative apps. The opt-out option has the potential to make the choices more realistic [34] by simulating a real-world context where individuals can exercise their right not to take up an app, given the apps on offer [20]. In our DCE, a participant had the option to choose or reject the hypothetical uptake of a smoking cessation app. However, when a participant selects the opt-out option, no information is provided on how they trade-off attribute levels or alternatives [13]. In some situations, a forced-choice scenario can be included, where participants who chose the opt-out option are prompted to make a choice regardless. An example of a scenario with an opt-out option is shown in Figure 2.
An example of a scenario with an opt-out option used in our DCE. DCE: discrete choice experiment.
Experimental Design
An experimental design is a systematic method of generating choice sets that are presented to respondents. This enables the specification of the choice sets that respondents see, with the objective of obtaining a high-quality data set [7]. When creating the experimental design, there are several aspects that need to be taken into consideration, including (1) the analytical model specification, (2) whether the aim is to estimate main effects only or interaction effects as well, (3) whether the design is labeled or unlabeled, (4) the number of choice tasks and blocking options to be used, (5) which type of design of the choice matrix to use (eg, full factorial or fractional factorial, orthogonal or efficient), and (6) how the attribute-level balance will be achieved. These are now considered.
Analytical Model Specification
The first step in the generation of an experimental design is to specify the analytical model to estimate the parameters of the DCE. This step is an important component of choosing the type of choice matrix design, described later in this paper. The approach selected here needs to be accounted for when generating the structure of the experimental design.
A discrete choice model describes the probability that an individual will choose a specific alternative. This probability is expressed as a function of measured attribute levels specific to the alternative and of characteristics of the individual making the choice. This probability is represented by the dependent variable (the choice variable), which indicates the choice made by participants [8]. In this modeling framework, the attributes are the independent variables [8,13].
As part of the analytical model specification, knowing what type of statistical analysis will be used is key. Data analysis involves regression modeling in a random utility framework [8]. The random utility model conventionally used is also based on the Lancaster theory of consumer demand [35], which together assume that individuals make trade-offs when making a decision and would choose an option that offers the greatest utility [36], determined by how much importance they place on the attributes associated with the product [37].
The multinomial logit (MNL) model has been previously described as the “workhorse” of DCE estimation [38,39], and it typically serves as a starting point for basic model estimation (although alternative models, such as probit, may be used). It is important to note that MNL requires some important assumptions and limitations—for example, independence of irrelevant alternatives, homogeneity of preferences, and independence of observed choices [40,41]. Extensions of MNL (eg, nested logit, mixed logit, and latent class models) may be used to account for these limitations [39,40].
Based on the model specified in our DCE, the underlying utility function for alternative j [38] is shown in Textbox 1.
The utility function used in our DCE research. DCE: discrete choice experiment.
1) U is the overall utility derived from alternative j.
2) β is the coefficient attached to Xj estimated in the analysis and represents the part-worth utility attached to each attribute level.
3) ε is the random error of the model—in other words, the unmeasured factors influencing the variation of preferences.
Main Effects or Interaction Effects
The next step in model specification is deciding whether main effects or interaction effects will be investigated. The main effects, the most commonly used, investigate the effect of each attribute level on the choice variable. The effect on the choice variable gained by combining two or more attribute levels (eg, app developer and the app's monthly cost) refers to an interaction effect [13]. In our DCE, given the novel nature of the research on the uptake of health apps and the lack of empirical evidence to suggest the presence of potential interactions between attributes, we decided to only look at main effects.
Labeled or Unlabeled Experiment
In a labeled experiment, the alternatives are specific and different (eg, smartphone app-based smoking cessation intervention vs website-based smoking cessation intervention) and alternative specific attributes could be used (eg, some attributes relevant only for apps and others for websites). This is in contrast to an unlabeled experimental design, where the alternatives are unspecified (eg, smoking cessation app alternative 1 vs smoking cessation app alternative 2) and also must have the same attributes. Given that a DCE model estimates parameters for each of the alternatives being considered, these alternative specific parameters must be included in the structure of the experimental design (described in the next section) in a labeled experiment; in an unlabeled experiment, because alternative specific parameters are arbitrary, they are excluded [22,42,43]. In health economics, the unlabeled approach is the most common. In our DCE, the unlabeled approach was deemed logical here as we were comparing different presentations of the same app. Therefore, our DCE design applied an unlabeled approach.
Generation of the Structure of the Experimental Design
Once the model is specified, the structure of the experimental design can be generated. For this stage, hypothetical alternatives are generated and combined to form choice tasks, based on the chosen attributes and their levels. Several different software packages may be used to generate the experimental design of a DCE, such as Ngene, SAS, SPEED, SPSS, and Sawtooth. For our DCE, Ngene software was used [44].
Number of Choice Tasks and Blocking
The next step in the generation of an experimental design is to decide on the choice task and blocking. To minimize respondent and cognitive burden, and the risk of participants losing interest during the DCE task, consideration must be paid to the target population, the number of tasks, and their complexity [13]. The higher the number of attributes, alternatives, and choice tasks, the higher the task complexity [20]. The literature suggests that a feasible limit is 18 choice sets per participant [45,46]. In the review by Marshall et al [27], most studies included between 7 and 16 choice sets. In our DCE, we administered 12 choice tasks per participant, which were deemed a number low enough to avoid excessive cognitive load but high enough to establish sufficient statistical precision.
We developed 48 choice tasks and blocked them into 4 survey versions (12 choice tasks for each). Each block represented a separate survey, and participants were randomly assigned to one of the four survey versions. Blocking is a technique widely used in DCEs to reduce cognitive burden by partitioning large experimental designs into subsets of equal size, thereby reducing the number of choice tasks that any one respondent is required to complete [47]. Blocks were generated in Ngene software, which allows for the minimization of the average correlation between the versions and attributes’ levels [48]. For the blocking to be successful, the number of choice tasks included in one block must be divisible by the number of attribute levels; in our DCE, attributes had either three or four levels.
It is noteworthy that to undertake the sample size calculation, it is crucial to know the number of alternatives per choice set, the largest number of levels of any attribute (for DCEs looking at main effects only) or the largest level of any two attributes (for a DCE looking at interaction effects), and the number of blocks [38]. Therefore, DCEs using blocking require a larger sample size [47].
Type of Choice Matrix Design
Depending on the number of attributes and their levels, a full- or fractional-factorial design can be applied. A full-factorial design would include all possible combinations of the attributes’ levels and allow the estimation of all main effects and interaction effects independent of one another [20]. However, this type of design is often considered impractical due to the high number of choice tasks required [20]. To illustrate this, the formula of calculation of the possible unique choice alternatives for a full-factorial design is LA, where L represents the number of levels and A the number of attributes [39]. If the attributes in the DCE have a different number of levels, these need to be calculated separately and multiplied together. To reduce response burden, in our DCE, we generated a fractional-factorial design in Ngene [44], representing a sample of possible alternatives from the full-factorial design. This way, we were able to reduce the total 432 alternatives in the full design (given by LA = 42 × 33) to a fractional sample of 96 alternatives, arranged in 48 choice pairs.
Systematic approaches for generation of fractional-factorial designs may be further categorized into orthogonal design and efficient design. An orthogonal design is a column-based design based on orthogonal arrays that present properties of orthogonality (attributes are statistically independent of one another) and level balance (levels of attributes appear an equal number of times) and does not introduce correlation between the attributes [38]. An orthogonal array is an optimal design that is often used for DCEs examining main effects when the number of attributes and their levels is small.
For studies with five or more attributes with two or more levels, an orthogonal design may not be practical. There has therefore been a recent change in thinking toward a nonorthogonal and statistically more efficient design [38]. When perfect orthogonality and balance cannot be achieved or are not desirable, an efficient design can be applied [20]. In contrast to an orthogonal design, an efficient design aims to increase the precision of parameter estimates for a given sample size (ie, minimizing the standard errors of the estimated coefficients), while allowing some limited correlation between attributes. The most widely used efficiency measure is the D-error, which may be easily estimated using various software packages, such as Ngene, and refers to the efficiency of the experimental design in extracting information from respondents [21]. Experimental designs generated using this approach are known as D-efficient designs. A D-efficient experimental design is also recommended to maximize statistical efficiency and minimize the variability of parameter estimates [7].
An efficient design requires that known prior information about the parameters (known as priors) be made available to the algorithm and also requires the analyst to specify the analytical model specification, as described previously. Depending on what information is available, one of three types of D-efficient design can be generated [21]:
Dz-efficient design (z stands for zero priors): If no prior information about the magnitude or directions of the parameters is available. Dz-efficient design is an orthogonal design. This design assumes the parameters are zero.
Dp-efficient design (p stands for priors): This assumes a fixed, certain value and direction for the parameters.
Db-efficient design (b stands for Bayesian): A Bayesian approach is whereby the parameter is not known with certainty but may be described by its probability distribution.
The best practice is to pilot the DCE. For the pilot phase, there is limited information available and using the Dz-efficient or Dp-efficient design is sensible. In our DCE, we chose to apply a Dp-efficient design, as the direction of priors of the app was known from the previously conducted survey, to narrow down the attribute levels and to provide prior estimates of the parameters for the attribute levels. For example, we knew that a trusted organization will likely positively influence uptake and cost estimated negatively so. The direction of priors was assumed to be a small near-zero negative or a positive value for the design.
The pilot phase provided the estimation that we used to generate a Db-efficient design for the final DCE. It is noteworthy that when the parameter priors are different from zero, the efficient design generated produces smaller prediction errors than orthogonal designs [21,49,50]. Hence, a D-efficient design will outperform an orthogonal design, and (given reliable priors), a Dp-efficient design will outperform a Dz-efficient design [21]. Further, when reasonable assumptions about the distributions are made, a Db-efficient design will outperform a Dp-efficient design. Therefore, it may be advisable to start piloting with a Dp-efficient design and to generate a Db-efficient design for the final DCE. The DCE literature provides a detailed and more comprehensive description of orthogonal and efficient designs [21] and the approximation of the Bayesian efficient design [23].
Attribute-Level Balance in the Model
The attribute-level balance aims to ensure all attribute levels ideally appear an equal number of times in the experimental design. The allocation of the attribute levels within the experimental design can affect statistical power; if a certain level is underrepresented in the choice sets generated, then the coefficient for that level cannot be easily estimated. How attributes levels are distributed is therefore an important consideration when designing the choice sets. Dominant alternatives, where all attribute levels of one alternative are more desirable than all attribute levels in the others, do not provide information about how trade-offs are made, as individuals usually would select the dominant alternatives. Therefore, avoiding dominant alternatives in the experimental design is important and can be achieved by consulting the software manual to ensure the correct algorithm is used. The syntax used in Ngene to generate choice sets of the pilot phase and more information about the algorithm used can be accessed on the Open Science Framework [19].
Piloting the DCE and Generating the Bayesian Design
In addition to providing estimations for the choice matrix design described above, piloting offers an opportunity to ensure that the information is presented clearly and that the choices are realistic and meaningful. It also provides insight into how cognitively demanding it is for respondents to complete. This can be achieved by gathering feedback on the survey completion process. The findings of the pilot may suggest that the DCE needs to be amended, such as reducing the number of choice sets or the number of attributes, so that the responses are a better reflection of the participants’ preferences and improve the precision in the parameter estimates [13].
There is no formal guidance on how large the pilot sample should be, and this is largely guided by the budget and complexity of the experimental design. Accuracy of the priors will improve with increasing sample size, but as few as 30 responses may be sufficient to generate useable data [44]. In our pilot study conducted with 49 individuals, feedback from the participants suggested that with the initial order of the attributes, there was a tendency to ignore the last two attributes, app description and images of the app, the most text-heavy attributes. This may have compromised the examination of the relative importance of those two attributes (app description and images of the app). Therefore, we decided to change the final order of the attributes from (1) monthly price of the app, (2) the ratings of the app, (3) who developed the app, (4) the description, and (5) images shown to the one listed in Figures 1 and 2. The longest completion time for the survey was under 12 min. Thus, we concluded that the number of choice tasks did not need to be reduced.
In our research, the data from the pilot phase were analyzed using the freely available Apollo package in R software [51]. The coefficients and their standard errors from the output were used as priors to generate the final choice sets using the Bayesian efficient design following the steps described previously. The syntax used in R used to analyze the pilot data and that used to generate the Bayesian efficient design in Ngene can be accessed on the Open Science Framework [19].
Internal Validity
Assessing the internal validity of a DCE can help with understanding the consistency and trade-off assumptions made by participants [52]. There are several ways to examine the internal validity of a DCE. For example, in the stability validity test, a choice task would be repeated later in the sequence to investigate the consistency of the participants’ decision, whether they would choose the same alternative [52]. Another way to test internal validity is the within-set dominated pairs type of internal validity, in which one alternative is a dominant alternative in which all attributes are the most desirable ones. The choice sets designed to measure internal validity are excluded from the analysis. There are several internal validity tests that are built into software packages such as MATLAB [52], although these can be produced manually as well. In our research, we used the stability validity test to check the internal validity by repeating a randomly generated choice task (in our case, it was the fourth). Therefore, participants were shown 12 choice tasks, plus an additional hold-out task. The data from the randomly generated hold-out task were excluded from the analysis.
Although internal validity checks provide some measure of data quality, it should be noted that answering a repeat choice inconsistently is not a violation of random utility theory [53]. Furthermore, there is no consensus on what to do with the data from responses that fail validity tests. Following the advice of Lancsar and Louviere [54], we did not exclude participants who failed the internal validity check, as that might have caused statistical bias or affected statistical efficiency. However, we reported data on internal validity to enable the reader to make a judgement on likely biases.
All additional study materials used in our example, including the full data set and the results of the DCE, can be accessed on Open Science Framework [19].
DiscussionSummary
This paper describes the development of a DCE, following the stages required to establish attributes and their levels, construct choice tasks, define the utility model, decide on labeled and unlabeled choices to apply, decide on the number of choice tasks that need to be generated, and make decisions on the structure of the experimental design, how to achieve attribute-level balance, how to assess the internal model validity, and how to pilot-test. In doing so, the intention is to advance methodological awareness of the application of stated preference methods in the field of digital health, as well as to provide researchers with an overview of their application using a case study of a DCE of smoking cessation app uptake.
Although DCEs are widely used to understand patient and provider choices in health care [8,10,15,55], they have only recently started to gain popularity in digital health [4-6] and as such represent an underused approach in digital health. With the growing evidence of the benefit of digital health initiatives, there are clear benefits to widening the application of DCEs so that they may more routinely inform digital health development, inform digital tool presentation, and, most importantly, predict uptake and engagement with digital products. Although several attempts have been made to measure engagement with digital tools using a wide range of methodologies [56-58], the insights we have from them that can be translated to uptake are limited. One plausible explanation is that uptake of digital tools is difficult to empirically measure.
Benefits and Limitations of DCEs
DCEs bring several benefits to help overcome the issue of measuring uptake in digital health or in other areas where the measurement of the predictors of uptake in a good or service is required. For example, as illustrated by the case study here, they enable the researcher to gain measurable insights into situations in which quantitative measures are hard to otherwise obtain, such as the factors impacting the uptake of health apps on curated health app portals. A DCE also helps to quantify preferences to support more complex decisions [59]. An example would be the consideration of how to plan the development of an app that would provide appealing looks or features that would promote uptake. The DCE methodology is also considered a convenient approach to investigate the uptake of new interventions, including digital health interventions [38], for example, digital behavior change interventions using a health and well-being smartphone app. Therefore, DCEs can be used in hypothetical circumstances, enabling the measurement of preferences for a potential policy change or digital health system change before it is implemented [13], such as the recent investigation of the uptake of a COVID-19 test-and-trace health app [3,4]. The experimental nature of the DCE also means that participants’ preferences can be recorded based on controlled experimental conditions, where attributes are systematically varied by researchers to obtain insight into the marginal effect of attribute changes on individuals’ choices [7].
Despite their benefits, the application of DCEs presents several challenges. As with all expressed preference methodologies, the hypothetical nature of the DCE choice set raises concerns about external validity and the degree to which real-world decisions might equate to those made by study participants under experimental conditions, a phenomenon known as the intention-behavior gap [60]. As such, participants may believe they would choose a scenario presented and described in a choice task, but in real life, there might be other factors that would influence their behaviors, such as the aesthetics of the app [28]. This limitation can at least partially be overcome by developing convincing and visually appealing choice tasks. Nevertheless, to date, there has been limited progress in testing for external validity due to the difficulty in investigating preferences in the real world [38]. Indeed, a recent systematic review of the literature on DCEs in health care reported that only 2% of the included studies (k=7) report details of the investigation of external validity [47], while an earlier systematic review and meta-analysis (k=6) found that DCEs have only a moderate level of accuracy in predicting behaviors of health choices [61]. To our knowledge, no study has been published that investigates the external validity of a DCE developed in digital health. One potential opportunity to undertake some testing would be through a curated health app portal, where the same health app is presented in two or more different ways. With the help of website analytics, actual user behavior could be measured in this situation.
A final significant concern associated with the use of a DCE is that any single choice set is unlikely to be able to present the user with all relevant attributes, regardless of how well it has been developed [61]. Choosing the most relevant attributes to test in a DCE, therefore, requires comprehensive preparatory research, which can lengthen the time required to undertake the development phase of any piece of work.
Conclusion
In summary, DCEs have significant potential in digital health research and can serve as an important decision-making tool in a field where observational data are lacking. We hope that the content of this paper provides a useful introduction and guide to those interested in developing such experiments in digital health.
AbbreviationsDCE
discrete choice experiment
MNL
multinomial logit
We are grateful to two experts in discrete choice experiments, Prof. Michiel Bliemer from the University of Sydney, Australia, a co-developer of Ngene software, and Prof. Stephane Hass from the University of Leeds, United Kingdom, a co-developer of the Apollo package in R software, for their advice on the syntax used to generate the choice tasks in Ngene and on the code used in the Apollo package. JAW and RC received funding from the National Institute for Health Research (NIHR) Applied Research Collaboration East of England. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care.
DS prepared the manuscript. All authors have reviewed the draft for important intellectual content and approved the final version.
JB has received unrestricted funding to study smoking cessation from Pfizer and J&J, who manufacture smoking cessation medications.
DannerMHummelJMVolzFvan ManenJGWiegardBDintsiosCBastianHGerberAIjzermanMJIntegrating patients' views into health technology assessment: analytic hierarchy process (AHP) as a method to elicit patient preferencesInt J Technol Assess Health Care2011102743697510.1017/S026646231100052322004779S0266462311000523HallJVineyRHaasMLouviereJUsing stated preference discrete choice modelling to evaluate health care programsJournal of Business Research2002065791026103210.1016/S0148-2963(02)00352-1WiertzCBanerjeeAAcarOAGhoshAPredicted adoption rates of contact tracing app configurations: insights from a choice-based conjoint study with a representative sample of the UK populationSSRN Journal2020043011910.2139/ssrn.3589199JonkerMde Bekker-GrobEVeldwijkJGoossensLBourSRutten-Van MölkenMCOVID-19 contact tracing apps: predicted uptake in the Netherlands based on a discrete choice experimentJMIR Mhealth Uhealth20201009810e207411510.2196/2074132795998v8i10e20741PMC7584977NittasVMütschMBraunJPuhanMASelf-monitoring app preferences for sun protection: discrete choice experiment survey analysisJ Med Internet Res202011272211e1888910.2196/1888933245282v22i11e18889PMC7732707LeighSAshall-PayneLAndrewsTBarriers and facilitators to the adoption of mobile health among health care professionals from the United Kingdom: discrete choice experimentJMIR Mhealth Uhealth2020070687e1770410.2196/1770432628118v8i7e17704PMC7381009Reed JohnsonFLancsarEMarshallDKilambiVMühlbacherARegierDABresnahanBWKanninenBBridgesJFPConstructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task ForceValue Health201316131310.1016/j.jval.2012.08.222323337210S1098-3015(12)04162-9SpinksJChaboyerWBucknallTTobianoGWhittyJAPatient and nurse preferences for nurse handover-using preferences to inform policy: a discrete choice experiment protocolBMJ Open20151111511e00894110.1136/bmjopen-2015-00894126560060bmjopen-2015-008941PMC4654394BridgesJFPHauberABMarshallDLloydAProsserLARegierDAJohnsonFRMauskopfJConjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task ForceValue Health2011061444031310.1016/j.jval.2010.11.01321669364S1098-3015(10)00083-5ClarkMDDetermannDPetrouSMoroDde Bekker-GrobEWDiscrete choice experiments in health economics: a review of the literaturePharmacoeconomics20140932988390210.1007/s40273-014-0170-x25005924KotnowskiKFongGTGallopel-MorvanKIslamTHammondDThe impact of cigarette packaging design among young females in Canada: findings from a discrete choice experimentNicotine Tob Res20160518513485610.1093/ntr/ntv11426014454ntv114LambooijMSHarmsenIAVeldwijkJde MelkerHMollemaLvan WeertYWMde WitGAConsistency between stated and revealed preferences: a discrete choice experiment and a behavioural experiment on vaccination behaviour comparedBMC Med Res Methodol20150312151910.1186/s12874-015-0010-52588789010.1186/s12874-015-0010-5PMC4359569ManghamLJHansonKMcPakeBHow to do (or not to do) ... designing a discrete choice experiment for application in a low-income countryHealth Policy Plan2009032421515810.1093/heapol/czn04719112071czn047Trapero-BertranMRodríguez-MartínBLópez-BastidaJWhat attributes should be included in a discrete choice experiment related to health technologies? A systematic literature reviewPLoS One2019147e021990510.1371/journal.pone.021990531318926PONE-D-18-28102PMC6639002HallJKennyPKingMLouviereJVineyRYeohAUsing stated preference discrete choice modelling to evaluate the introduction of varicella vaccinationHealth Econ2002071154576510.1002/hec.69412112494FiebigDGKnoxSVineyRHaasMStreetDJPreferences for new and existing contraceptive productsHealth Econ20110920 Suppl 1355210.1002/hec.168621809412Terris-PrestholtFHansonKMacPhailCVickermanPReesHWattsCHow much demand for new HIV prevention technologies can we really expect? Results from a discrete choice experiment in South AfricaPLoS One2013812e8319310.1371/journal.pone.008319324386160PONE-D-13-30009PMC3875434Terris-PrestholtFQuaifeMVickermanPParameterising user uptake in economic evaluations: the role of discrete choice experimentsHealth Econ20160225 Suppl 11162310.1002/hec.329726773825PMC5066644SzinayDRoryCJonesAWhittyJChadbornTBrownJNaughtonFAdult Smokers' Preferences for the Uptake of Smoking Cessation Apps: A Discrete Choice Experiment202103122020-11-01https://osf.io/5439x/LancsarELouviereJConducting discrete choice experiments to inform healthcare decision making: a user's guidePharmacoeconomics20082686617710.2165/00019053-200826080-00004186204602684RoseJMBliemerMCJConstructing efficient stated choice experimental designsTransport Rev20090929558761710.1080/01441640902827623de Bekker-GrobEWHolLDonkersBvan DamLHabbemaJDFvan LeerdamMEKuipersEJEssink-BotMSteyerbergEWLabeled versus unlabeled discrete choice experiments in health economics: an application to colorectal cancer screeningValue Health20101323152310.1111/j.1524-4733.2009.00670.x19912597S1098-3015(10)60378-6BliemerMCRoseJMHessSApproximation of bayesian efficiency in experimental choice designsJ Choice Model2008119812610.1016/s1755-5345(13)70024-1HauberABGonzálezJMGroothuis-OudshoornCGMPriorTMarshallDACunninghamCIJzermanMJBridgesJFPStatistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task ForceValue Health2016061943001510.1016/j.jval.2016.04.00427325321S1098-3015(16)30291-1KløjgaardMEBechMSøgaardRDesigning a stated choice experiment: the value of a qualitative processJ Choice Model20125211810.1016/s1755-5345(13)70050-2BuchananJBlairEThomsonKLOrmondroydEWatkinsHTaylorJCWordsworthSDo health professionals value genomic testing? A discrete choice experiment in inherited cardiovascular diseaseEur J Hum Genet20191111271116394810.1038/s41431-019-0452-z3118654610.1038/s41431-019-0452-zPMC6870981MarshallDBridgesJFPHauberBCameronRDonnalleyLFyieKJohnsonFRConjoint analysis applications in health: how are studies being designed and reported? An update on current practice in the published literature between 2005 and 2008Patient20101201342495610.2165/11539650-000000000-000002227343201312067-201003040-00006SzinayDJonesAChadbornTBrownJNaughtonFInfluences on the uptake of and engagement with health and well-being smartphone apps: systematic reviewJ Med Internet Res20200529225e175722310.2196/1757232348255v22i5e17572PMC7293059SzinayDPerskiOJonesAChadbornTBrownJNaughtonFInfluences on the uptake of health and well-being apps and curated app portals: think-aloud and interview studyJMIR Mhealth Uhealth2021042794e2717310.2196/2717333904827v9i4e27173PMC8114158RakotonarivoOSSchaafsmaMHockleyNA systematic review of the reliability and validity of discrete choice experiments in valuing non-market environmental goodsJ Environ Manage201612011839810910.1016/j.jenvman.2016.08.03227576151S0301-4797(16)30570-9RatcliffeJLongworthLInvestigating the structural reliability of a discrete choice experiment within health technology assessmentInt J Technol Assess Health Care2002181139144YangJReedSDHassSSkeenMBJohnsonFRIs easier better than harder? An experiment on choice experiments for benefit-risk tradeoff preferencesMed Decis Making2021024122223210.1177/0272989X2097983333463397MühlbacherAJohnsonFRChoice experiments to quantify preferences for health and healthcare: state of the practiceAppl Health Econ Health Policy2016061432536610.1007/s40258-016-0232-72699238610.1007/s40258-016-0232-7WatsonVBeckerFde Bekker-GrobEDiscrete choice experiment response rates: a meta-analysisHealth Econ2017062668101710.1002/hec.335427122445RyanMGerardKAmaya-AmayaMRyanMGerardKAmaya-AmayaMDiscrete choice experiments in a nutshellUsing Discrete Choice Experiments to Value Health and Health Care2008Dordrecht, the NetherlandsSpringer1346McFaddenDZarembkaPConditional logit analysis of qualitative choice behaviorFrontiers in Econometrics1973Cambridge, MAAcademic Press10542PotoglouDBurgePFlynnTNettenAMalleyJForderJBrazierJEBest-worst scaling vs. discrete choice experiments: an empirical comparison using social care dataSoc Sci Med201105721017172710.1016/j.socscimed.2011.03.02721530040S0277-9536(11)00190-0de Bekker-GrobEWRyanMGerardKDiscrete choice experiments in health economics: a review of the literatureHealth Econ2012022121457210.1002/hec.169722223558HensherDARoseJMGreenWApplied Choice Analysis: A Primer2005Cambridge, UKCambridge University Press1HensherDAGreeneWHThe mixed logit model: the state of practiceTransportation20033021337610.1023/A:1022558715350TrainKLogit. Discrete Choice Methods with Simulation. 2 ed2009CambridgeCambridge University Press3475KruijshaarMEEssink-BotMDonkersBLoomanCWSiersemaPDSteyerbergEWA labelled discrete choice experiment adds realism to the choices presented: preferences for surveillance tests for Barrett esophagusBMC Med Res Methodol20090519913110.1186/1471-2288-9-31194540221471-2288-9-31PMC2695479JinWJiangHLiuYKlampflEDo labeled versus unlabeled treatments of alternatives' names influence stated choice outputs? Results from a mode choice studyPLoS One2017128e017882610.1371/journal.pone.017882628806764PONE-D-17-16130PMC5555680ChoiceMetricsNGene.1.1.1 User Manual & Reference Guide A2012OnlineChoiceMetrics1ChristofidesNJMuirheadDJewkesRKPenn-KekanaLConcoDNWomen's experiences of and preferences for services after rape in South Africa: interview studyBMJ2006012833275352091310.1136/bmj.38664.482060.5516330476bmj.38664.482060.55PMC1352051HansonKMcPakeBNakambaPArchardLPreferences for hospital quality in Zambia: results from a discrete choice experimentHealth Econ20050714768770110.1002/hec.95915619273SoekhaiVde Bekker-GrobEWEllisARVassCMDiscrete choice experiments in health economics: past, present and futurePharmacoeconomics2019023722012610.1007/s40273-018-0734-23039204010.1007/s40273-018-0734-2PMC6386055JanssenEMHauberABBridgesJFPConducting a discrete-choice experiment study following recommendations for good research practices: an application for eliciting patient preferences for diabetes treatmentsValue Health201801211596810.1016/j.jval.2017.07.00129304942S1098-3015(17)30320-0FerriniSScarpaRDesigns with a priori information for nonmarket valuation with choice experiments: a Monte Carlo studyJ Environ Econ Manage200755333426310.1016/j.jeem.2006.10.007KesselsRJonesBGoosPVandebroekMLAn efficient algorithm for constructing Bayesian optimal choice designsSSRN J2006110.2139/ssrn.968620HessSPalmaDApollo: a flexible, powerful and customisable freeware package for choice model estimation and applicationJ Choice Model2019093210017010.1016/j.jocm.2019.100170JohnsonFRYangJReedSDThe internal validity of discrete choice experiment data: a testing tool for quantitative assessmentsValue Health2019022221576010.1016/j.jval.2018.07.87630711059S1098-3015(18)33233-9HessSDalyABatleyRRevisiting consistency with random utility maximisation: theory and implications for practical workTheory Decis20181284218120410.1007/s11238-017-9651-7319837839651PMC6953994LancsarELouviereJDeleting 'irrational' responses from discrete choice experiments: a case of investigating or imposing preferences?Health Econ20060815879781110.1002/hec.110416615039QuaifeMTerris-PrestholtFEakleRCabrera EscobarMAKilbourne-BrookMMvunduraMMeyer-RathGDelany-MoretlweSVickermanPThe cost-effectiveness of multi-purpose HIV and pregnancy prevention technologies in South AfricaJ Int AIDS Soc201803213110.1002/jia2.2506429537654PMC5851344PerskiOBlandfordAGarnettCCraneDWestRMichieSA self-report measure of engagement with digital behavior change interventions (DBCIs): development and psychometric evaluation of the "DBCI Engagement Scale"Transl Behav Med202002031012677710.1093/tbm/ibz039309273575423758PMC8411853Craig LefebvreRTadaYHilfikerSWBaurCThe assessment of user engagement with eHealth content: the eHealth Engagement ScaleJ Comput Mediat Commun201007011546668110.1111/j.1083-6101.2009.01514.xO'BrienHLTomsEGThe development and evaluation of a survey to measure user engagementJ. Am. Soc. Inf. Sci20091019611506910.1002/asi.21229Brett HauberAFairchildAOReed JohnsonFQuantifying benefit-risk preferences for medical interventions: an overview of a growing empirical literatureAppl Health Econ Health Policy2013081143192910.1007/s40258-013-0028-y23637054AjzenIThe theory of planned behaviorOrgan Behav Hum Decis Process19911250217921110.1016/0749-5978(91)90020-tQuaifeMTerris-PrestholtFDi TannaGLVickermanPHow well do discrete choice experiments predict health choices? A systematic review and meta-analysis of external validityEur J Health Econ20181119810536610.1007/s10198-018-0954-62938022910.1007/s10198-018-0954-6