Published on 16.12.16 in Vol 18, No 12 (2016): December
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/5927, first published May 03, 2016.
IDEAS (Integrate, Design, Assess, and Share): A Framework and Toolkit of Strategies for the Development of More Effective Digital Interventions to Change Health Behavior
Developing effective digital interventions to change health behavior has been a challenging goal for academics and industry players alike. Guiding intervention design using the best combination of approaches available is necessary if effective technologies are to be developed. Behavioral theory, design thinking, user-centered design, rigorous evaluation, and dissemination each have widely acknowledged merits in their application to digital health interventions. This paper introduces IDEAS, a step-by-step process for integrating these approaches to guide the development and evaluation of more effective digital interventions. IDEAS is comprised of 10 phases (empathize, specify, ground, ideate, prototype, gather, build, pilot, evaluate, and share), grouped into 4 overarching stages: Integrate, Design, Assess, and Share (IDEAS). Each of these phases is described and a summary of theory-based behavioral strategies that may inform intervention design is provided. The IDEAS framework strives to provide sufficient detail without being overly prescriptive so that it may be useful and readily applied by both investigators and industry partners in the development of their own mHealth, eHealth, and other digital health behavior change interventions.
J Med Internet Res 2016;18(12):e317
Digital technology has rapidly and dramatically shifted how humans interact with the world and presents an unprecedented opportunity to develop, test, and widely disseminate effective health behavior change interventions. The prospect of modifying lifestyle behaviors, such as diet, physical activity, smoking, and sleep, to improve health outcomes has increasingly driven efforts in both academia and industry. However, achieving meaningful and sustained improvements in health behaviors has eluded researchers and industry players alike, and numerous challenges remain. Within industry, most digital health interventions are yet to incorporate theory-based strategies known to drive changes in health behaviors or undergo systematic testing to demonstrate their effectiveness [- ]. Moreover, interventions are often plagued by rapidly declining retention rates [ ], with a quarter of downloaded health apps used only once and three-quarters discontinued after the tenth use [ ]. Within academia, interventions are more often grounded in behavioral theory and tested for their efficacy [ ]. However, they face similar challenges around declining retention rates, and the rapid pace of new technology development makes it increasingly difficult for researchers to develop, pilot-test, and evaluate interventions before such technologies become outdated or obsolete [ , ]. In addition, researcher-driven technologies often do not benefit from numerous rapid cycles of fine-tuning based on user feedback nor do they usually become widely disseminated among the broader public [ ].
In light of these challenges, investigators have called for digital health interventions to be (1) grounded in behavioral theory [- ], (2) grounded in an in-depth qualitative understanding of the target population [ ], (3) rapidly and iteratively developed with multiple stages of user feedback [ , ], (4) subject to rigorous evaluation [ , ], and (5) widely disseminated [ ]. However, published frameworks to guide the development of such technologies are disparate and no single approach integrates these elements. In addition, specific recommendations for integrating principles from behavioral theory are lacking in available frameworks. These gaps limit our ability to advance health behavior change research and practice. As our knowledge evolves in this young field, investigators have been called on to publish the methods they use to develop interventions to further advance the field [ ]. This paper responds to that call and aims to build on prior models by combining behavioral theory, design thinking, and intervention evaluation and dissemination through a systematic framework to guide the development of more effective digital health interventions. It introduces Integrate, Design, Assess, and Share (IDEAS), an integrated 10-phase process.
Essential Components of a Framework
To maximize the potential efficacy of interventions to change health behavior, those who design interventions should have an understanding of theory or the hypothesized mechanisms underlying human behavior and behavior change . Behavioral theory is widely acknowledged to be critical to the development of interventions to change health behavior [ - , - ] and increasing evidence suggests that health interventions grounded in theory are more effective than those without such theoretical foundations [ ]. Among the most frequently used theories in health behavior research are social cognitive theory, the health belief model, and the transtheoretical model [ , ]. One approach to intervention development is to use one or more theories to identify the key constructs to be targeted in the intervention. For example, the health belief model suggests that a behavior change intervention should target perceived susceptibility, severity, benefits, and barriers. An alternative approach is to select behavior change techniques (eg, from those listed in the taxonomy of behavior change techniques [BCTs] [ ]) and use these to construct the intervention. In principle, the two approaches could be used in combination, selecting techniques that are believed based on theory and/or evidence to be likely to produce change in the targeted behaviors.
A recent consensus statement on the prevention of noncommunicable diseases emphasized the importance of using human-centered design, or design thinking, to develop effective and innovative interventions . First coined by David Kelley, a founder of the design firm IDEO and the Stanford University Hasso Plattner Institute of Design (d.school), design thinking is intended to guide the development of more creative and innovative solutions [ ]. The process has been summarized by the Stanford d.school as encompassing 5 phases: “empathize” (understand target population), “define” (identify goals and scope), “ideate” (brainstorm potential solutions), “prototype” (mock up primitive potential solutions), and “test” (gather feedback from target users) [ ]. Design thinking starts by reframing the context for behavioral change around “what matters most” to a target group rather than “what’s the matter” with them [ ]. This approach enables the design of more empathetic solutions that are more desirable to target populations. The importance of grounding behavioral health interventions in a deep understanding of a target population has been emphasized by experts in public health and health psychology [ , ]. Design thinking next involves rapidly and iteratively brainstorming, prototyping, and gathering user feedback on potential solutions. This process is similar to user-centered design, first presented by Norman and Draper [ ], and involves gathering user feedback throughout intervention development, refining designs through prototyping and iteration, and including multidisciplinary skills and perspectives [ ]. Core to design thinking is the notion that everyone has the potential to be highly creative and can learn to apply design-based approaches such as need-finding, brainstorming, prototyping, and iteration to unlock their creative potential [ ]. It has been suggested that design thinking may increase the self-efficacy of those who practice the process [ ] and frees even the most novice designers to generate more creative ideas and solutions. As a result, the process has been increasingly adopted among a diverse range of academics and industry players [ ] and has been adapted to better serve the varied contexts in which it is used, particularly through discipline-specific frameworks and processes [ , ].
Evaluation and Dissemination
There have long been calls for the evaluation and dissemination of digital health interventions . Rigorous evaluation is essential to judge whether or not an intervention achieves its desired effect. Published evaluations also contribute to the literature and evidence base, helping uncover the potential and challenges of using digital interventions to improve health outcomes [ , ]. Rigorous evaluation methods also enable more reliable conclusions. It has been recommended that the purpose of evaluation, the balance of potential benefits and risks, and the level of resource investment required by providers and users can help drive the necessary level of evidence needed [ ]. As the UK Medical Research Council emphasizes, evaluation efforts should employ randomization whenever possible because it is considered the most robust approach to preventing several forms of bias. Without randomization to the intervention or a control or comparison condition, it is not possible to determine whether the intervention itself was responsible for the observed effects or whether a selected group of participants, for instance, happened to be highly motivated and might have improved without the intervention [ ].
Moreover, dissemination is crucial if digital health interventions are to fulfill their potential. One of the great promises of digital interventions is their ability to reach broad segments of the population with minimal cost . However, most publicly available apps to promote dietary behavior change have not been assessed in rigorous randomized controlled efficacy trials [ , ]. In turn, most digital health interventions that have demonstrated efficacy in peer-reviewed trials are not available to the public; instead, they have been created ad hoc for research purposes [ ]. Thus, interventions that are publicly available have not been evaluated and those that have been evaluated are not publicly available [ ]. As a result, there is a need for effective interventions to be more widely disseminated to populations that may benefit.
Existing Frameworks and Limitations
Using the best combination of recommended approaches to guide intervention design is important if effective technologies to change health behavior are to be developed. Researchers often rely on published frameworks to guide them through the process of designing digital interventions . However, currently available frameworks are numerous, disparate, and do not fully integrate behavioral theory, design thinking, and evaluation and dissemination. Although many electronic health (eHealth) frameworks exist, most envision their objective as guiding the development of technologies to facilitate medical or patient care (eg, patient-physician communication, access to medical records) rather than to modify health behavior [ , ]. For instance, Van Velsen et al [ ] propose a “requirements development” approach in which stakeholder interviews are meant to lead directly to a list of technical specifications to be developed (eg, one-stop portal for patient information). This approach may be appropriate for building products to facilitate logistics or care provision in medical settings, but designing for behavior change is a different type of endeavor that requires thoughtful integration of behavioral theory and evidence.
Relatively few frameworks focus on guiding the development of digital interventions for the express purpose of changing health behavior. Among the frameworks that do are Yardley et al’s  person-based approach, Ludden et al’s [ ] design research perspective, and Brown et al’s [ ] health information technology usability evaluation model. Although each of these approaches provides valuable guidance for investigators, each focuses on a particular aspect of intervention development and none provides guidance on behavioral strategies that may be used in intervention design. Hekler et al’s [ ] process, referred to as behavioral science-informed user experience design, combines a user-centered design approach with the use of behavioral theory-driven strategies. This process notably suggests the integration of user-centered and theory-based approaches, but it does not provide step-by-step guidance on how others may replicate the approach. Whittaker and colleagues [ ] have proposed perhaps the most comprehensive step-by-step framework to date which involves 5 phases (focus groups, pretesting, pilot, randomized controlled trial [RCT], interviews) to guide overall mobile health intervention development and evaluation. Whittaker’s framework has been applied to the iterative development of numerous mobile health interventions [ - ], includes stages for user feedback and evaluation, and states the importance of using behavioral theory. However, it does not make use of design thinking approaches such as ideation, brainstorming, or rapid prototyping, nor does it include specific guidance on behavioral strategies that may inform intervention design. Although it has been suggested that design thinking and behavioral science can together inform the development of more effective digital health interventions [ ], no published frameworks appear to combine behavioral theory, design thinking, and evaluation and dissemination into a comprehensive step-by-step process for guiding digital interventions to change health behavior. This gap limits our ability to advance health behavior change research and practice.
To address the need for a framework that more fully integrates strengths from behavioral theory, design thinking, and evaluation and dissemination, we introduce IDEAS, a framework to better guide the development of digital health interventions to change behavior. IDEAS was informed by a multisector team of researchers, designers, and engineers, and was then applied to and refined in the iterative development of Vegethon, a mobile health (mHealth) intervention that demonstrated user acceptability and initial efficacy . IDEAS consists of 10 phases: (1) empathize with target users, (2) specify target behavior, (3) ground in behavioral theory, (4) ideate implementation strategies, (5) prototype potential products, (6) gather user feedback, (7) build a minimum viable product, (8) pilot test to assess potential efficacy and usability, (9) evaluate efficacy in an RCT, and (10) share intervention and findings. These phases are grouped into 4 overarching categories: Integrate, Design, Assess, and Share ( ).
IDEAS aims to facilitate the creation of more effective interventions by using strengths from a combination of disciplines at the intersection of digital health. The framework builds on design thinking and user-centered design in that it is iterative, engages multidisciplinary perspectives through a cross-sector team, includes stages for ideation and prototyping (phases 4, 5, 7), and integrates user insights throughout (phases 1 and 6). It focuses the design process around defining behavioral goals grounded in evidence (phase 2) and encourages the inclusion of theory-driven behavioral strategies both initially (phase 3) and throughout the design process. IDEAS integrates evaluation methods captured by the UK Medical Research Council framework  by emphasizing the importance of rigorously evaluating behavioral outcomes through both pilot and more substantially powered RCTs (phases 8 and 9). Finally, given the abundance of low-quality interventions currently available to the public, IDEAS concludes with a stage for dissemination (phase 10), which capitalizes on the readily scalable nature of digital interventions to provide access to target populations that may benefit. Dissemination also includes sharing findings with other audiences that can use the findings to advance the field and science of behavior change.
The 10 IDEAS phases are summarized in a step-by-step fashion in. As with design thinking, although the IDEAS phases are presented sequentially, they are not necessarily intended to be carried out linearly, and projects are encouraged to loop back through the phases in an iterative fashion as different ideas and directions are explored and refined [ ].
Phase 1: Empathize With Target Users
The first 3 IDEAS phases (empathize, specify, and ground) aim to integrate insights from users and behavioral theory and orient the intervention development process around a measurable target behavior. These phases are primarily about information gathering to help inform the development in later stages of a more well-accepted, theory-driven intervention.
In phase 1, qualitative research is undertaken to gain a deeper understanding of the selected target population and their needs [, ]. This need-finding stage can include observations, interviews, focus groups, and questionnaires [ ]. Designers gain insight into users’ unmet or latent needs, which they may not necessarily be aware of or be able to articulate [ ]. The goal of this stage is to move beyond identifying incremental improvements that users might be able to articulate (eg, refinements to a health app they already enjoy using) and to instead uncover deeper needs, values, and motivations that may help inspire more innovative and creative solutions [ ]. For instance, nuanced insights, such as not wanting to feel like a failure or lacking family support for healthier cooking, while not directly suggestive of potential intervention solutions, may equip team members with the background and context necessary to develop an intervention that helps users feel supported with a virtual social support group and positive language. By engaging all members of the intervention development team in this qualitative research stage, it may be possible to guide the development of solutions that are more relevant and acceptable to the target population [ ].
Phase 2: Specify Target Behavior
Insights gathered from users help inform the next phase, in which a highly specific target behavior is selected. This target behavior defines both the purpose of the intervention as well as the outcomes by which the intervention will be judged. For instance, an initially broad intervention goal of “increase physical activity” may be refined to “take 10,000 steps each day.” The narrowing of the intervention goal in this way helps to focus the scope of idea generation and has been associated with success in the context of highly innovative concepts .
Multiple, and sometimes competing, factors may inform the selection of the target behavior. Insights from users in the previous phase may help investigators identify appropriate potential target behaviors that would be well accepted by individuals. A review of the literature may be conducted to identify the health benefits of possible target behaviors under consideration. For example, before deciding whether to target greater vegetable consumption or greater breakfast consumption, it would be advantageous to understand whether one behavior is more likely to have a greater health impact than the other. Some behaviors may be more susceptible to change than others, particularly if they are readily countable and therefore easier for users to self-monitor and modify. Some behaviors may also have the potential to produce a beneficial cascade effect, improving other health behaviors or aspects of the same health behavior . Qualitative investigations have also shown that users may be more interested in technologies that frame behaviors as actions to increase as opposed to decrease (eg, increasing the number of days per week free of sugar-sweetened beverages vs cutting back on sugar-sweetened beverages) [ ]. Thus, principles that may guide the selection of the intervention target behavior include (1) a behavior that is well accepted by the target population, (2) an evidence base demonstrating a significant health benefit to changing the target behavior, (3) a behavior that is highly specific and countable and therefore more actionable, (4) a behavior that is framed as something to increase versus decrease, and (5) optimally a behavior that if changed has the potential for producing a beneficial cascade effect on related and reinforcing behaviors.
Phase 3: Ground in Behavioral Theory
Once a target behavior is identified, strategies are explored to ground intervention development in behavioral theory. The intervention design team may seek to gain familiarity with a range of theory-driven behavioral strategies available for inclusion. Strategies best suited to the target behavior and intervention delivery medium are identified for possible inclusion. Researchers may draw from behavioral theories, such as social cognitive theory , which are frequently used in academia to guide mobile health interventions [ ]. Researchers may also draw from collections of theory-driven behavioral strategies, which have increasingly been introduced by investigators. For example, Michie et al’s [ ] taxonomy summarizes 93 behavior change techniques such as goal setting, self-monitoring, feedback, prompts/cues, and action planning.
In our cumulative experience, we have also found it useful to apply a theory-driven process motivation lens during intervention development . Process motivators, first introduced by Robinson [ ], may be used to make the process itself of behavior change more engaging and intrinsically rewarding. Process motivators stand in contrast to outcome motivators, which focus on the eventual outcomes of behavior change such as weight loss, physical appearance, becoming more fit, and reducing risks of future chronic diseases [ ]. Although outcome motivators have historically dominated medical and public health interventions (eg, smoking cessation interventions emphasizing reduced risk of lung cancer), such rational appeals are limited in their motivational power to initiate and sustain behavior changes [ , ]. Outcome motivators rely on results that are often delayed and difficult to achieve and maintain, reducing their motivational impact and an individual’s self-efficacy for behavior change [ , , ]. By contrast, interventions that emphasize motivation for participating in the intervention itself, or the process of behavior change, may be more effective in initiating and sustaining behavior changes [ ]. It has been suggested that process motivators, such as fun, taste, pride, choice, and challenge, can make the process of eating healthier or of engaging in physical activity more inherently enjoyable and desirable [ ]. presents examples of behavioral strategies using process motivation, adapted and extended from Robinson [ ], and based on research on intrinsic motivation and interventions to change behavior [ , - ], which may be useful to investigators in the development of their own interventions.
|Challenge||Maintain optimal levels of moderate challenge (ie, not too hard, not too easy)|
|Choice/control||Provide objective and perceived choice and control over one’s environment and actions|
|Community||Provide social meaning (public recognition, identification with desirable group) for accomplishments|
|Competence||Provide immediate, frequent, clear, constructive, encouraging positive feedback following success|
|Competition||Facilitate social comparison and competition among individuals, groups, or teams|
|Context||Embed intervention into real/imaginary contexts with stories/characters|
|Curiosity||Provide sensory (color, taste, sound) and cognitive (mystery) curiosity and surprise|
|Growth mindset||Cultivate belief that behaviors/preferences (eg, for foods, activity levels) are malleable with effort|
|Identity||Facilitate an identity shift related to the behavior change (eg, someone who is now a runner)|
|Personalization||Personalize intervention using an individual’s name and personally relevant content|
|Pride||Cultivate pride and a sense of accomplishment|
|Piggybacking||Engage individuals in social movements (eg, animal rights) to harness deeper values|
|Reframing||Cast the purpose of a behavior in a more positive light to improve thoughts or feelings about it|
|Taste||Emphasize the taste and texture of healthier foods|
|Teamwork||Facilitate cooperation and teamwork among individuals, groups, or teams|
Regardless of which behavioral theories or strategies are used or preferred by investigators, the goal of this phase is to consider ways in which behavioral theory can be incorporated into intervention development. Equipped with behavioral theory and strategies, subsequent stages enable the creative translation of such strategies into tangible and specific intervention components. Theory-driven insights can also be instrumental in helping guide the interpretation of user interview findings and addressing gaps in user understanding and awareness. At this stage, the team may wish to explore, discuss, compare, and reconcile insights from users and behavioral strategies in order to prepare for the brainstorming phase.
Phase 4: Ideate Creative Implementation Strategies
The next 4 phases (ideate, prototype, gather, and build) involve a highly iterative design process that focuses idea generation around the target behavior and takes insights from users and behavioral theory into account throughout.
To begin, a series of group brainstorming sessions is held to ideate potential intervention solutions, components, and features. The agreed-upon target behavior focuses the scope of idea generation  and the aim is to generate a large number of diverse ideas that could influence the target behavior. Previously gathered user insights and behavioral theories help inspire the range of ideas. For instance, a “self-monitoring” behavioral strategy in the context of a target behavior to “increase daily steps” might inspire an idea to engage users in taking a photograph of the outdoor scenery each time they go for a run. User insights suggesting that individuals prefer outdoor running to other types of activities might lend further credibility to such an idea. Hundreds of ideas may be generated “ranging from the absurd to the obvious” (p 31 [ ]). Adhesive notes can be used to capture different ideas (one idea per note), which can be displayed on a wall or board, and visual representations of ideas are encouraged to facilitate communication of ideas to other team members [ ]. Divergent thinking is achieved through interdisciplinary teams engaging in structured brainstorming sessions. Team members are encouraged to generate ideas that build on prior ideas and/or are divergent from those already suggested. Deferring judgment is a central rule during this phase to encourage rather than discourage idea generation [ ]. Throughout the process, weaker ideas drop off early on, whereas stronger ideas “naturally rise to the top” [ ]. This phase is ideally undertaken with a cross-sector team to enable the generation of greater numbers of more varied and creative ideas. As brainstorming progresses, discussing and debating ideas during this highly creative stage is recommended because it has been associated with more novel innovations and produces a more thorough exploration of possible solutions [ ].
Phase 5: Prototype Potential Products
Prototypes are then created to share and discuss ideas with team members and to facilitate both further ideation (phase 4) and the rapid gathering of user feedback (phase 6). The prototyping phase is exemplified by the adage “enlightened trial and error outperforms the planning of flawless intellect” (p 1 ). Prototypes may be sketched, hand crafted using primitive materials, and/or developed into clickable mockups of digital interfaces. In the earliest stages, low-cost, low-fidelity (ie, “quick and dirty”) prototypes are developed rapidly to gather feedback from users early and often. This approach allows intervention designers to quickly and cheaply gather feedback on many different possible intervention approaches before investing significant resources in any one particular approach or suboptimal solution [ , ]. Experimental studies have demonstrated that the act of developing multiple prototypes in parallel (vs sequentially) leads to objectively stronger results [ ]. In one experiment, when novice designers were instructed to develop multiple prototypes (vs a single prototype) before obtaining user feedback, designs overall tended to be more divergent and final prototypes were superior as measured by click-through data and blinded expert ratings. These findings were explained by qualitative data suggesting that parallel prototyping reduced fixation on a particular idea and encouraged designers to instead explore multiple directions before optimizing in any one direction. By contrast, sequential prototyping implicitly encouraged the refinement of the initial prototype at the expense of exploration of more divergent alternatives [ ]. Although time constraints often lead teams to focus on the realization of a single idea rather than the iteration of multiple ideas [ ], prototype iteration has been shown to lead to objectively stronger final products even when a team is under time constraints [ ].
Phase 6: Gather User Feedback
User feedback is then gathered on prototypes. Methods may include informal conversations, usability tests, surveys, and in-depth qualitative interviews. For instance, during a usability test of an early prototype, a researcher may observe participants use the intervention and participants may be asked to think aloud by providing commentary during the process. These methods of inquiry seek to uncover users’ interests in the overall product and eagerness to use it as part of their typical routine, potential impediments to usage on a regular basis, suggestions for improvement, and unexpected new ideas or opportunities. When this phase is initially employed in the design process, the focus is on quickly gathering initial user impressions and inspiring further divergent ideation. As this phase is repeated in quick succession with the ideate and prototype phases such that the phases inform one another , the goal is increasingly to inform concept refinement and focus. Thus, the ideate, prototype, and gather phases are carried out in a fluid and iterative fashion until a more refined design solution is reached [ ].
Phase 7: Build a Minimum Viable Product
Next, a fully functional minimum viable product (MVP) is developed to facilitate early learning from users. This stage focuses on a level of detail beyond the prototype stage with decisions made regarding user experience, visual design, content, and logic (eg, to calculate graphs, to trigger push notifications). Analytics are built into the product to enable the collection of a rich dataset capturing patterns of usage. The use of analytics such as metadata may be applied in the development of just-in-time, adaptive interventions (JITAIs), in which real-time data are used to tailor an intervention to the dynamically changing needs of a user [, ]. The practical challenges and realities of building a digital intervention surface numerous decisions that need to be made with the technical developers. A multisector team can include perspectives to help ensure that the influence of behavioral theory, user needs and desires, and technical, financial, and practical feasibility are properly balanced and reconciled throughout the decision-making process. A core tenet of the MVP concept is to avoid wasting precious time and resources on perfecting a product that may be substantially altered in subsequent phases based on user feedback [ ]. The goal, therefore, is to quickly develop only a minimum version of the product necessary to facilitate pilot testing among users, without incorporating additional unnecessary features. However, this industry-driven tenet must be balanced with the common assumption among behavioral scientists that many behavioral strategies are most effective when used in concert, rather than in isolation [ ], which may encourage the inclusion of more behavioral strategies. Weighing these opposing priorities (ie, fewer features vs more theory-driven strategies) will determine how theory- and feature-rich the minimum viable intervention becomes at this stage. The degree to which an intervention is theory- and feature-rich will vary in each case and depend on the unique resource (eg, financial, human, time) constraints as well as the behavioral needs of the intervention. As with all phases in IDEAS, this phase can be conducted iteratively with other phases, particularly phase 6, to gather user feedback to help inform product decisions before pilot testing.
Phase 8: Pilot Test
The final 3 phases (pilot, evaluate, and share) aim to assess and disseminate the final intervention and evaluation results. This process begins by subjecting the MVP to pilot testing to gather information on the usability, feasibility, and potential efficacy of the first viable version of the intervention [, , ]. For example, usability and satisfaction questionnaires may ask individuals on a 5-point scale how strongly they agree with statements such as “I have found this app easy to use,” “this product has motivated me to be more physically active,” or “I have had a hard time remembering to use this product” [ ]. Analysis of product usage data helps identify usage patterns and peak usage times, which may inform notification timing. Qualitative and usability interviews lend insight into preferred features, problematic or confusing user flows, and opportunities for intervention modifications or improvements. Study design and implementation procedures, such as recruitment, enrollment, retention, and data collection, are tested and refined to avoid problems in subsequent larger-scale evaluation studies [ ]. Randomization, which may [ ] or may not [ ] be used at this pilot stage, enables the potential efficacy of the intervention to be assessed [ ]. Based on the wealth of information gathered during this stage, the intervention is further refined until it is ready for a larger-scale evaluation.
Phase 9: Evaluate Efficacy
A sufficiently powered RCT is then conducted to test intervention efficacy. RCTs are considered the most robust study design for evaluating interventions due to their ability to minimize bias [, , ]. Outcomes may include measures of the original target health behavior; mediators, proximal behaviors, or conditions that may lie on the causal pathway to behavior change; and broader effects on related health behaviors and risk factors. Assessing possible moderators, or preexisting factors that may help define characteristics of users that respond more or less to the intervention, has also been recommended to better define the appropriate target audience for ultimate dissemination [ ]. Given the rapidly evolving nature of mobile technology and the time involved in carrying out a rigorous evaluation study, the digital intervention may need to be continuously refined during the trial to ensure that it does not become obsolete by the time the trial is completed [ ]. Researchers need to weigh the advantages of such an approach with the potential limitations to internal validity that are introduced if the intervention is modified too dramatically. To yield reliable and valid results, several best practices for conducting an RCT are followed. Randomization is used to eliminate conscious and unconscious selection bias in the assignment of participants to the intervention versus a control condition [ ]. Proper randomization is achieved when the allocation sequence is unpredictable and concealed, such that the researcher enrolling participants does not know in advance which treatment the next participant will receive [ , ]. The widely recommended “intention-to-treat” analysis approach is used to preserve the benefits of randomization, in which all randomized participants are included in the analysis and retained in the original groups to which they were allocated, regardless of adherence to the digital intervention, missing data, or dropouts [ , ]. Data collectors and researchers who perform statistical analyses are blinded to intervention assignment to prevent the introduction of biases in the interpretation or analysis of data. For further guidance on RCTs, researchers may consult the evidence-based Consolidated Standards of Reporting Trials (CONSORT) 2010 guidelines [ ].
Phase 10: Share Widely
There are two types of sharing implied in the IDEAS framework. First, learnings and evaluation results should be disseminated to both researchers and digital health intervention developers to help advance the field and improve the effectiveness of future interventions. For example, the evaluation trial’s findings should be published, including a description of the final intervention and its theoretical basis and/or use of behavior change techniques, such as those using standardized language and descriptors from a taxonomy of behavior change techniques . The second type of sharing involves disseminating efficacious interventions more broadly with appropriate target audiences. One of the greatest promises of digital health interventions is their potential to be widely scaled to users at a relatively low cost [ ]. Thus, once intervention efficacy has been demonstrated, the intervention is disseminated to the broader population that may benefit the most, as demonstrated in the evaluation. Scaling an intervention for wide dissemination is not a trivial undertaking. Health-related intervention goals may be less salient in certain dissemination settings where, for instance, health care providers, health insurers, and/or employers may be more motivated by cost savings. The systems architecture (eg, servers, databases, storage systems) of an intervention may need to be adjusted to account for greater traffic at scale. Further intervention refinements may be warranted based on dissemination goals [ ]. Cost considerations, including marketing, maintenance, and ongoing implementation, may significantly influence adoption [ ]. Academic researchers may wish to form strategic partnerships with relevant public and private sector organizations to support the more effective and sustainable dissemination of the final intervention [ ].
How to Apply IDEAS
Engage a Multidisciplinary Team
Digital health is increasingly acknowledged to be an inherently transdisciplinary endeavor in which involving user-centered designers, programmers, behavioral researchers (particularly those working in the area of behavioral intervention development), and industry partners is crucial [, ]. The effectiveness of digital health interventions rests on multiple diverse factors including esthetic design, behavioral theory, evidence grounding, user centeredness, technical capacity, and demonstrated efficacy. Only a true multidisciplinary team will have the knowledge and expertise to address these complex factors in tandem [ ]. Engaging all team members throughout the full intervention development process may help the team proceed through each of the phases iteratively, nimbly, collaboratively, and with greater buy-in at all stages. This practice can also help ensure that all team members understand users’ needs and the theoretical grounding underpinning the intervention. Finally, enlisting the full multidisciplinary team may facilitate the generation of more highly divergent ideas and prototypes, which has been shown to lead to more creative and efficacious final intervention designs [ ].
Use a Flexible, Iterative Approach
As with design thinking, the teams that stand to benefit the most from applying the IDEAS framework are those who will apply the stages flexibly, pursuing multiple stages in parallel, in combination, more than once, and/or iteratively, as new directions are explored and refined . For example, some high-performing teams have been shown to combine, in particular, the ideate, prototype, and gather phases, rapidly shifting between them as needed. A single brainstorming session (ideate phase) may flow directly into the use of highly primitive prototypes (prototype phase) to communicate ideas to team members, generate new ideas, and/or gather user feedback (gather phase) to facilitate further brainstorming (ideate phase) [ ]. In this way, the 10 IDEAS phases are more like “a system of overlapping spaces, rather than a sequence of orderly steps” as design thinking has been described [ ].
Transition Between Divergence and Convergence
A key consideration in applying the framework involves balancing the opposing processes of divergent and convergent thinking. Idea divergence, or the generation of highly diverse and varied ideas, is critical during early stages of the intervention development process to fully consider the range of user experiences and possible solutions. However, idea convergence, or the narrowing of possible ideas, becomes equally important to identify a target behavior and refine potential solutions without being hampered by a continual revision of prior decisions which may impede progress. High-performing teams have been shown to shift their behavior nimbly and repeatedly as needed throughout the design process, exhibiting greater debate and divergence during “concept generation” stages and less debate and more focused attention during “concept selection” stages . In a space where digital technology is young and the best solutions are likely yet to be uncovered, a highly divergent approach in brainstorming possible solutions may maximize the likelihood of identifying the most potent solutions. As the design process progresses, successful teams may find it advantageous to transition from divergence to convergence, particularly once a set of possible solutions have been identified for refinement [ ].
This paper introduces IDEAS, a framework to guide the design, development, and evaluation of digital interventions to change health behavior. It includes a summary of behavioral strategies that may be useful for intervention developers seeking to apply IDEAS in the development of their own interventions. IDEAS is among the few frameworks that aim to guide the development of digital interventions to change health behavior. It builds on Whittaker et al’s  framework by incorporating phases for ideation and prototyping that may contribute to more creative interventions and by providing guidance on behavioral strategies that may inform more effective technologies. It draws on strengths from Yardley et al’s [ ] person-based approach through a focus on qualitative inquiry and Hekler et al’s [ ] behavioral science-informed user experience design model through a focus on behavioral strategies and user-centered design. Although the importance of combining behavioral theory and design thinking has been emphasized [ ], IDEAS appears to be the first framework to provide step-by-step guidance on integrating these approaches.
Among the strengths of IDEAS is its provision of a toolkit of behavioral strategies grounded in process motivation, which may aid and inspire intervention design. These strategies include theory-driven approaches not presently captured by the taxonomy of behavior change techniques  and may serve as a useful additional resource for intervention developers seeking to develop their own solutions. Moreover, theory by nature is abstract [ ] and intervention developers may not know how to translate such insights into concrete intervention features. The IDEAS framework fills this gap by suggesting that ideating, prototyping, and gathering user feedback may be used iteratively and in quick succession to incrementally translate theories into highly relevant and practical intervention components. IDEAS also addresses the topic of creativity, which has been given considerably less attention in the mHealth and eHealth design literature [ ]. Through structured brainstorming sessions and rapid prototyping with user feedback, intervention developers aiming to incorporate greater amounts of creativity into their designs are provided with practical guidance on how to do so. In addition, by integrating user insights throughout the intervention development process, IDEAS helps guard against an unexpected mismatch between intervention goals and user goals, which may lead to low user satisfaction and poor adherence to an intervention [ , ]. Finally, the promise of digital interventions lies in their potential for reaching broad segments of the population [ ]; where other frameworks have neglected dissemination altogether, the inclusion of dissemination as a key phase may help more intervention developers take this stage into consideration to advance the field overall and produce greater population impacts on health.
Several limitations to this framework exist. Despite research exploring the application of design-based approaches among novice designers , less-experienced users may find it difficult to effectively apply some of the suggested methods. There are practical challenges inherent to working with a multidisciplinary team and team members may disagree in a counterproductive manner or find it difficult to advance potential intervention solutions. The behavioral strategies presented are not exhaustive. The framework does not suggest the randomized evaluation of isolated intervention components [ ]. However, it is commonly argued that behavioral strategies work best in concert with one another; thus, evaluating individual intervention components separately may not necessarily be advantageous in this context [ ]. This framework is not the only approach to intervention development and many other useful approaches may complement or advance IDEAS. Future research is recommended to systematically evaluate the IDEAS framework.
The IDEAS framework is proposed to guide the design, development, and rigorous evaluation of more creative and effective digital health interventions. It integrates behavioral theory, design thinking and user-centered design, and evaluation and dissemination, and summarizes a list of theory-driven behavioral strategies that may be useful to intervention developers. IDEAS is intended to accelerate the translation of behavioral theory and evidence into industry practice where most digital health interventions are born. Other researchers who use IDEAS or alternative frameworks are encouraged to report on their processes and outcomes so that we and others may learn from their experiences and continue to improve the quality, efficacy, and effectiveness of our digital health interventions.
This research was supported by a Gates Cambridge Scholarship (SAM), Award Number R01DK0091831 from the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) of the National Institutes of Health (NIH) (CDG), an unrestricted gift from the Nutrition Science Initiative (NuSI) (CDG), and a grant from the Stanford Child Health Research Institute (TNR). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIDDK or NIH. The authors would like to thank Katherine Dotter, Dustin Yoder, Jae Berman, Ryan Swanson, Maya Mathur, and Haley Schmoyer for helping inform the development of the framework.
SAM developed the framework and wrote the manuscript. SS, TNR, CDG, and ACK contributed guidance and consultation throughout and provided feedback on the manuscript. All authors read and approved the final manuscript.
Conflicts of Interest
- Azar KM, Lesser LI, Laing BY, Stephens J, Aurora MS, Burke LE, et al. Mobile applications for weight management: theory-based content analysis. Am J Prev Med 2013 Nov;45(5):583-589. [CrossRef] [Medline]
- Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl Behav Med 2011 Mar;1(1):53-71 [FREE Full text] [CrossRef] [Medline]
- Pagoto S, Schneider K, Jojic M, DeBiasse M, Mann D. Evidence-based strategies in weight-loss mobile apps. Am J Prev Med 2013 Nov;45(5):576-582. [CrossRef] [Medline]
- Chen J, Cade JE, Allman-Farinelli M. The most popular smartphone apps for weight loss: a quality assessment. JMIR Mhealth Uhealth 2015;3(4):e104 [FREE Full text] [CrossRef] [Medline]
- McLean V. Motivating patients to use smartphone health apps. PRWeb. 2011 Apr 25. URL: http://www.prweb.com/releases/2011/04/prweb5268884.htm [accessed 2016-06-26] [WebCite Cache]
- Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth 2015;3(1):e20 [FREE Full text] [CrossRef] [Medline]
- Baker TB, Gustafson DH, Shah D. How can research keep up with eHealth? Ten strategies for increasing the timeliness and usefulness of eHealth research. J Med Internet Res 2014;16(2):e36 [FREE Full text] [CrossRef] [Medline]
- Robinson T, Patrick K, Eng T, Gustafson D. An evidence-based approach to interactive health communication: a challenge to medicine in the information age. Science Panel on Interactive Communication and Health. JAMA 1998;280(14):1264-1269.
- Pagoto S, Bennett GG. How behavioral science can advance digital health. Transl Behav Med 2013 Sep;3(3):271-276 [FREE Full text] [CrossRef] [Medline]
- Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res 2015;17(1):e30 [FREE Full text] [CrossRef]
- Tate EB, Spruijt-Metz D, O'Reilly G, Jordan-Marsh M, Gotsis M, Pentz MA, et al. mHealth approaches to child obesity prevention: successes, unique challenges, and next directions. Transl Behav Med 2013 Dec;3(4):406-415 [FREE Full text] [CrossRef] [Medline]
- Hekler E, King A, Banerjee B, Robinson T, Alonso M, Cirimele J. A case study of BSUED: behavioral science-informed user experience design. In: Proceedings of the SIGCHI Conference Extended Abstracts on Human Factors in Computing Systems. 2011 Presented at: CHI 2011; May 7–12, 2011; Vancouver, BC p. 7-12.
- Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236 [FREE Full text] [CrossRef] [Medline]
- Bennett GG, Glasgow RE. The delivery of public health interventions via the Internet: actualizing their potential. Annu Rev Public Health 2009;30:273-292. [CrossRef] [Medline]
- Whittaker R, Merry S, Dorey E, Maddison R. A development and evaluation process for mHealth interventions: examples from New Zealand. J Health Commun 2012 May;17 Suppl 1:11-21. [CrossRef] [Medline]
- Davis R, Campbell R, Hildon Z, Hobbs L, Michie S. Theories of behaviour and behaviour change across the social and behavioural sciences: a scoping review. Health Psychol Rev 2015 Sep;9(3):323-344 [FREE Full text] [CrossRef] [Medline]
- Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ 2000 Sep 16;321(7262):694-696 [FREE Full text] [Medline]
- Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, et al. Designing and evaluating complex interventions to improve health care. BMJ 2007 Mar 3;334(7591):455-459 [FREE Full text] [CrossRef] [Medline]
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655 [FREE Full text] [Medline]
- Painter JE, Borba CP, Hynes M, Mays D, Glanz K. The use of theory in health behavior research from 2000 to 2005: a systematic review. Ann Behav Med 2008 Jun;35(3):358-362. [CrossRef] [Medline]
- Glanz K, Bishop DB. The role of behavioral science theory in development and implementation of public health interventions. Annu Rev Public Health 2010;31:399-418. [CrossRef] [Medline]
- Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med 2013 Aug;46(1):81-95. [CrossRef] [Medline]
- Matheson GO, Klügl M, Engebretsen L, Bendiksen F, Blair SN, Börjesson M, et al. Prevention and management of noncommunicable disease: the IOC Consensus Statement, Lausanne 2013. Clin J Sport Med 2013 Nov;23(6):419-429. [CrossRef] [Medline]
- Kelley T, Kelley D. Creative Confidence: Unleashing the Creative Potential Within All of Us. New York: Crown Business; 2013.
- Stanford d.school: Design Thinking Bootcamp Bootleg. 2011. URL: http://dschool.stanford.edu/wp-content/uploads/2011/03/BootcampBootleg2010v2SLIM.pdf [accessed 2016-11-22] [WebCite Cache]
- Norman D, Draper S. User-centred system design: new perspective on human-computer interaction. Hillsdale, NJ: Lawrence Earlbaum Associate; 1986.
- Hermawati S, Lawson G. Managing obesity through mobile phone applications: a state-of-the-art review from a user-centred design perspective. Pers Ubiquit Comput 2014;18(8):2003-2023.
- Martin R. Design of Business: Why Design Thinking is the Next Competitive Advantage. Boston: Harvard Business Press; 2009.
- Dow SP, Glassco A, Kass J, Schwarz M, Schwartz DL, Klemmer SR. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Trans Comput-Hum Interact 2010 Dec 01;17(4):1-24. [CrossRef]
- Katoppo M, Sudradjat I. Combining Participatory Action Research (PAR) and Design Thinking (DT) as an alternative research method in architecture. Procedia Soc Behav Sci 2015 May;184:118-125. [CrossRef]
- Quade S, Schlueter O. Adapting design thinking for media prototyping: innovative collaboration at universities and workplaces. In: ICERI2014 Proceedings. 2014 Nov 17 Presented at: 7th International Conference of Education, Research and Innovation; Nov 17-19, 2014; Seville, Spain p. 5221-5229.
- Fiordelli M, Diviani N, Schulz PJ. Mapping mHealth research: a decade of evolution. J Med Internet Res 2013;15(5):e95 [FREE Full text] [CrossRef] [Medline]
- Coughlin SS, Whitehead M, Sheats JQ, Mastromonico J, Hardy D, Smith SA. Smartphone applications for promoting healthy diet and nutrition: a literature review. Jacobs J Food Nutr 2015;2(3):21 [FREE Full text] [Medline]
- Horvath KJ, Ecklund AM, Hunt SL, Nelson TF, Toomey TL. Developing Internet-based health interventions: a guide for public health researchers and practitioners. J Med Internet Res 2015;17(1):e28 [FREE Full text] [CrossRef] [Medline]
- van Gemert-Pijnen JE, Nijland N, van LM, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011;13(4):e111 [FREE Full text] [CrossRef] [Medline]
- Van Velsen L, Wentzel J, Van Gemert-Pijnen JE. Designing eHealth that matters via a multidisciplinary requirements development approach. JMIR Res Protoc 2013;2(1):e21 [FREE Full text] [CrossRef] [Medline]
- Ludden GD, van Rompay TJ, Kelders SM, van Gemert-Pijnen JE. How to increase reach and adherence of web-based interventions: a design research viewpoint. J Med Internet Res 2015;17(7):e172 [FREE Full text] [CrossRef] [Medline]
- Brown W, Yen P, Rojas M, Schnall R. Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology. J Biomed Inform 2013 Dec;46(6):1080-1087 [FREE Full text] [CrossRef] [Medline]
- Fjeldsoe BS, Miller YD, O'Brien JL, Marshall AL. Iterative development of MobileMums: a physical activity intervention for women with young children. Int J Behav Nutr Phys Act 2012;9:151 [FREE Full text] [CrossRef] [Medline]
- Waterlander W, Whittaker R, McRobbie H, Dorey E, Ball K, Maddison R, et al. Development of an evidence-based mHealth weight management program using a formative research process. JMIR Mhealth Uhealth 2014;2(3):e18 [FREE Full text] [CrossRef] [Medline]
- Whittaker R, Dorey E, Bramley D, Bullen C, Denny S, Elley CR, et al. A theory-based video messaging mobile phone intervention for smoking cessation: randomized controlled trial. J Med Internet Res 2011;13(1):e10 [FREE Full text] [CrossRef] [Medline]
- Whittaker R, Merry S, Stasiak K, McDowell H, Doherty I, Shepherd M, et al. MEMO--a mobile phone depression prevention intervention for adolescents: development process and postprogram findings on acceptability from a randomized controlled trial. J Med Internet Res 2012;14(1):e13 [FREE Full text] [CrossRef] [Medline]
- Mummah SA, Mathur M, King AC, Gardner CD, Sutton S. Mobile technology for vegetable consumption: a randomized controlled pilot study in overweight adults. JMIR Mhealth Uhealth 2016 May;4(2):e51 [FREE Full text] [CrossRef] [Medline]
- Brown T, Wyatt J. Design thinking for social innovation. Stanford Soc Innov Rev 2010;30(5):29-53.
- Thondoo M, Strachan DL, Nakirunda M, Ndima S, Muiambo A, Källander K, et al. Potential roles of Mhealth for community health workers: formative research with end users in Uganda and Mozambique. JMIR Mhealth Uhealth 2015;3(3):e76 [FREE Full text] [CrossRef] [Medline]
- Hammersley M. Reading Ethnographic Research: A Critical Guide. London: Longman; 1990.
- Peeters M, Megens C, Hummels C, Brombacher A. Experiential probes: probing for emerging behavior patterns in everyday life. 2013 Aug 26 Presented at: 2013 IASDR Conference: Consilience and Innovation in Design; Aug 26-30, 2013; Tokyo, Japan.
- Gasparini A. Perspective and use of empathy in design thinking. 2015 Presented at: The Eighth International Conference on Advances in Computer-Human Interactions; Feb 22-27, 2015; Lisbon, Portugal.
- Lynna GS, Akgunb AE. Project visioning: its components and impact on new product success. J Prod Innov Manag 2001 Nov;18(6):374-387. [CrossRef]
- Duhigg C. The power of habit: why we do what we do in life and business. New York: Random House Trade Paperbacks; 2012.
- Dennison L, Morrison L, Conway G, Yardley L. Opportunities and challenges for smartphone applications in supporting health behavior change: qualitative study. J Med Internet Res 2013;15(4):e86 [FREE Full text] [CrossRef] [Medline]
- Bandura A. Social Foundations of Thought and Action. Englewood Cliffs, NJ: Prentice-Hall; 1986.
- Robinson T. Stealth interventions for obesity prevention and control: motvating behavior change. In: Dubé L, Bechara A, Dagher A, Drewnowski A, Lebel J, James P, editors. Obesity Prevention: The Role of Brain and Society on Individual Behavior. New York: Elsevier Inc; 2010:319-327.
- Wansink B. Mindless Eating: Why We Eat More Than We Think. New York: Bantam Books; 2006.
- Ariely D. Predictably Irrational: The Hidden Forces that Shape our Decisions. New York: HarperCollins; 2008.
- Bandura A. Self-Efficacy: The Exercise of Control. New York: Freeman and Company; 1997.
- Lepper M, Master A, Yow W. Intrinsic motivation in education. In: Maehr ML, Karabenick SA, Urdan TC, editors. Advances in Motivation and Achievement, Volume 15: Social Psychological Perspectives. Bingley: Emerald; 2008:521-555.
- Robinson TN, Borzekowski DLG. Effects of the SMART classroom curriculum to reduce child and family screen time. J Communication 2006 Mar;56(1):1-26. [CrossRef]
- Robinson TN. Save the world, prevent obesity: piggybacking on existing social and ideological movements. Obesity (Silver Spring) 2010 Feb;18 Suppl 1:S17-S22 [FREE Full text] [CrossRef] [Medline]
- Dweck C. Mindset: The New Psychology of Success. New York: Ballantine Books; 2008.
- Crum AJ, Langer EJ. Mind-set matters: exercise and the placebo effect. Psychol Sci 2007 Feb;18(2):165-171. [CrossRef] [Medline]
- Petrie K, Cameron L, Ellis C, Buick D, Weinman J. Changing illness perceptions after myocardial infarction: an early intervention randomized controlled trial. Psychosom Med 2002;64(4):580-586.
- Kelley T, Littman J. The Ten Faces of Innovation: IDEO's Strategies for Defeating the Devil's Advocate and Driving Creativity Throughout Your Organization. New York: Random House; 2005.
- Pelled LH, Eisenhardt KM, Xin KR. Exploring the black box: an analysis of work group diversity, conflict, and performance. Admin Sci Quart 1999 Mar;44(1):1. [CrossRef]
- Dow S, Heddleston K, Klemmer S. The efficacy of prototyping under time constraints. In: Proceedings of the Seventh ACM Conference on Creativity and Cognition. 1640260: ACM; 2009 Presented at: C&C '09: Seventh ACM Conference on Creativity and Cognition; Oct 27-30, 2009; Berkeley, CA p. 165-174.
- Vredenburg K, Isensee S, Righi C. User-Centered Design: An Integrated Approach. Upper Saddle River, NJ: Prentice Hall; 2002.
- Schrage M. Serious Play: How the World's Best Companies Simulate to Innovate. Brighton: Harvard Business School Press; 1999.
- Seidel V, Fixon S. Adopting design thinking in novice mutlidisciplinary teams: the application and limits of design methods and reflexive practices. J Prod Innov Manage 2013;30:19-33.
- Spruijt-Metz D, Nilsen W. Dynamic models of behavior for just-in-time adaptive interventions. Lect Notes Comput Sc 2014;13(3):13-17.
- Nahum-Shani S, Smith S, Tewari A, Witkiewitz K, Collins L, Spring B. Just-in-Time Adaptive Interventions (JITAIs): An Organizing Framework for Ongoing Health Behavior Support (Technical Report No. 14-126). University Park, PA: The Methodology Center; 2014:1-37.
- Ries E. The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. New York: Crown Business; 2011.
- King AC, Hekler EB, Grieco LA, Winter SJ, Sheats JL, Buman MP, et al. Harnessing different motivational frames via mobile phones to promote daily physical activity and reduce sedentary behavior in aging adults. PLoS One 2013;8(4):e62613 [FREE Full text] [CrossRef] [Medline]
- Eldridge SM, Ashby D, Feder GS, Rudnicka AR, Ukoumunne OC. Lessons for cluster randomized trials in the twenty-first century: a systematic review of trials in primary care. Clin Trials 2004 Feb 01;1(1):80-90. [CrossRef]
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. Int J Nurs Stud 2013 May;50(5):587-592. [CrossRef] [Medline]
- Nollen NL, Mayo MS, Carlson SE, Rapoff MA, Goggin KJ, Ellerbeck EF. Mobile technology for obesity prevention: a randomized pilot study in racial- and ethnic-minority girls. Am J Prev Med 2014 Apr;46(4):404-408 [FREE Full text] [CrossRef] [Medline]
- Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J Clin Nurs 2003 Jan;12(1):77-84. [Medline]
- Kraemer HC, Frank E, Kupfer DJ. Moderators of treatment outcomes: clinical, research, and policy importance. JAMA 2006 Sep 13;296(10):1286-1289. [CrossRef] [Medline]
- Schulz KF. Randomized controlled trials. Clin Obstet Gynecol 1998 Jun;41(2):245-256. [Medline]
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995 Feb 1;273(5):408-412. [Medline]
- Schulz KF, Chalmers I, Grimes DA, Altman DG. Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA 1994 Jul 13;272(2):125-128. [Medline]
- Herman A, Botser IB, Tenenbaum S, Chechick A. Intention-to-treat analysis and accounting for missing data in orthopaedic randomized clinical trials. J Bone Joint Surg Am 2009 Sep;91(9):2137-2143. [CrossRef] [Medline]
- Hollis S, Campbell F. What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ 1999 Sep 11;319(7211):670-674. [CrossRef]
- Moher D, Hopewell S, Schulz K, Montori V, Gøtzsche P, Devereaux P. CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol 2010;63(8):e1-e37. [CrossRef]
- Schneider F, Schulz DN, Pouwels LH, de Vries H, van Osch LA. The use of a proactive dissemination strategy to optimize reach of an internet-delivered computer tailored lifestyle intervention. BMC Public Health 2013;13:721 [FREE Full text] [CrossRef] [Medline]
- Gersick CJ. Marking time: predictable transitions in task groups. Acad Manage J 1989 Jun 01;32(2):274-309. [CrossRef]
- Kelders SM, Van Gemert-Pijnen JE, Werkman A, Nijland N, Seydel ER. Effectiveness of a Web-based intervention aimed at healthy dietary and physical activity behavior: a randomized controlled trial about users and usage. J Med Internet Res 2011;13(2):e32 [FREE Full text] [CrossRef] [Medline]
- Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med 2007 May;32(5 Suppl):S112-S118 [FREE Full text] [CrossRef] [Medline]
|d.school: Stanford University Hasso Plattner Institute of Design|
|eHealth: electronic health|
|IDEAS: Integrate, Design, Assess, and Share|
|JITAI: just-in-time, adaptive intervention|
|mHealth: mobile health|
|MVP: minimum viable product|
|RCT: randomized controlled trial|
Edited by G Eysenbach; submitted 03.05.16; peer-reviewed by G Munro, A Atienza, M Peeples; comments to author 16.06.16; revised version received 25.09.16; accepted 12.10.16; published 16.12.16
©Sarah Ann Mummah, Thomas N Robinson, Abby C King, Christopher D Gardner, Stephen Sutton. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 16.12.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.