JMIR Publications

Select Journals for Content Updates

When finished, please click submit.

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

This paper is in the following e-collection/theme issue:

    Viewpoint

    Tailored and Integrated Web-Based Tools for Improving Psychosocial Outcomes of Cancer Patients: The DoTTI Development Framework

    1Priority Research Centre for Health Behaviour, University of Newcastle & Hunter Medical Research Institute, Callaghan, Australia

    2School of Electrical Engineering and Computer Science, Faculty of Engineering and Built Environment, University of Newcastle, Callaghan, Australia

    3Department of Haematology, Royal North Shore Hospital, University of Sydney, St Leonards, Australia

    Corresponding Author:

    Rochelle Smits, BPsych(Hons)

    Priority Research Centre for Health Behaviour

    University of Newcastle & Hunter Medical Research Institute

    HMRI Building

    University of Newcastle

    Callaghan, 2308

    Australia

    Phone: 61 2 4042 0700

    Fax:61 2 4042 0044

    Email:


    ABSTRACT

    Background: Effective communication with cancer patients and their families about their disease, treatment options, and possible outcomes may improve psychosocial outcomes. However, traditional approaches to providing information to patients, including verbal information and written booklets, have a number of shortcomings centered on their limited ability to meet patient preferences and literacy levels. New-generation Web-based technologies offer an innovative and pragmatic solution for overcoming these limitations by providing a platform for interactive information seeking, information sharing, and user-centered tailoring.

    Objective: The primary goal of this paper is to discuss the advantages of comprehensive and iterative Web-based technologies for health information provision and propose a four-phase framework for the development of Web-based information tools.

    Methods: The proposed framework draws on our experience of constructing a Web-based information tool for hematological cancer patients and their families. The framework is based on principles for the development and evaluation of complex interventions and draws on the Agile methodology of software programming that emphasizes collaboration and iteration throughout the development process.

    Results: The DoTTI framework provides a model for a comprehensive and iterative approach to the development of Web-based informational tools for patients. The process involves 4 phases of development: (1) Design and development, (2) Testing early iterations, (3) Testing for effectiveness, and (4) Integration and implementation. At each step, stakeholders (including researchers, clinicians, consumers, and programmers) are engaged in consultations to review progress, provide feedback on versions of the Web-based tool, and based on feedback, determine the appropriate next steps in development.

    Conclusions: This 4-phase framework is evidence-informed and consumer-centered and could be applied widely to develop Web-based programs for a diverse range of diseases.

    J Med Internet Res 2014;16(3):e76)

    doi:10.2196/jmir.2849

    KEYWORDS



    Introduction

    Global Burden of Cancer and its Psychosocial Consequences

    Cancer is one of the leading causes of death world-wide, accounting for 7.6 million deaths in 2008 [1]. A diagnosis of cancer can impose a significant psychological burden on patients and their families. Challenges include coping with uncertainty surrounding prognosis, making important treatment decisions, and learning how to manage often debilitating physical, psychological, and social effects of the disease. Cancer care is complex, involving a multidisciplinary team, including general practitioners, cancer doctors, nurses, and other allied health professionals, and patients often have to travel for treatment [2]. Between 32% and 48% of individuals diagnosed with cancer experience psychological distress, including anxiety and depression [3]. Failure to address these issues through the provision of appropriate information and psychosocial care may have a significant impact on clinical patient outcomes and the health care system, including higher frequency and intensity of physical symptoms [4], poorer adherence to treatment regimes [5], and increased utilization of medical services [6].

    The Importance of Effective Communication in Reducing Psychosocial Burden

    Effective communication and provision of information to cancer patients and their families about their disease, treatment options, and possible outcomes improve psychosocial outcomes [7,8]. Recognition of the importance of effective communication has been driven by increased consumer activism, as well as legal imperatives to ensure patients are well informed of their treatment options and are able to exercise control over their role in making decisions regarding their care [9]. In order to make an informed decision, for example, a patient must be provided with clear and sufficient information about the risks and benefits of available treatment options [10]. Failing to fully inform patients about their condition, treatment options, and potential consequences, as well as misrepresenting information, can lead to legal challenges and medical litigation [11]. This has led to a shift from paternalistic approaches to information provision and disclosure within the health care system to a model that emphasizes autonomy and patient-centered care [12], which are reflected in changes to legislation, bioethical guidelines, and accepted principles within the medical profession [12-15].

    The philosophy of patient-centered care promotes self-management and patient empowerment by emphasizing patients as partners in decision making and offering tailored health care that is responsive to patient needs [16]. Effective patient-centered care is associated with improved health outcomes, enhanced patient and practitioner satisfaction, and decreased use of health care services [16]. The Institute of Medicine report, “Crossing the Quality Chasm”, promoted patient-centered care as an essential component of quality health care [12], a position that is passionately supported by consumer advocates. The involvement of consumers in health service reform and research is essential to ensuring patients’ needs, values, and preferences are represented [17]. The health consumer movement has strengthened in recent decades with the establishment of consumer advocacy groups, including MacMillan (United Kingdom) [18], Cancer Voices (Australia) [19], and LiveStrong (United States) [20]. Consumers are actively involved in campaigns to engender change in health care policy and practice, creating awareness of diseases and offering support to patients, and determining research priorities and distribution of funding [21]. The sense of autonomy and assertiveness promoted by the health consumer movement has led to an appeal by patients for active involvement in decision making in partnership with their health care provider [22].

    Accounting for Individual Patient Variation in Communication

    Ensuring patients are well informed and able to participate in making complex decisions about their care is complicated by several factors. As well as providing patients with information, there is a need to ensure that information is provided in a way that is understood and recalled. Health literacy refers to an individual’s ability to obtain, process, and understand health information to make appropriate decisions about their health care [23]. Around one in five adults in the United States has low health literacy [23]. Individuals with low health literacy often face challenges in acting as fully informed consumers. Consequently, they may lack a clear understanding about their condition, their options for and the potential consequences of treatment [24], and therefore find it difficult to be active in making decisions about their health care [23]. Given the increasingly complex treatment options available to cancer patients, health literacy must be an important consideration when providing information to patients [23].

    Accounting for Clinician Variation and Patient Preferences in Communication

    Patients often report that clinicians do not provide information about their diagnosis and treatment options in ways they can understand [23]. This mismatch between the information provided by clinicians and the patient’s level of health literacy hinders the ability of patients to recall and utilize the information [23]. There is also significant variation in patient preferences for involvement in medical decision making [25]. Degner and colleagues found that 22% of breast cancer patients preferred to make the decision about their treatment, 44% wanted a collaborative approach between themselves and their clinician, and 34% of women preferred their clinician having responsibility for treatment decisions [25]. Achieving the desired level of involvement in decision making may be further hindered by the ability of clinicians to accurately assess patient preferences. Several studies have revealed that clinician perceptions are incongruent with cancer patient preferences for involvement in medical decision making in approximately 58% of cases [25,26].


    The Limitations of Traditional Approaches to Information Provision

    There are a number of existing approaches to providing information to patients. These include patient-clinician interaction, written/printed leaflets and booklets, audio-visual materials, and more recently, websites accessed via the Internet. However, these approaches have a number of limitations. For example, patients consistently report being dissatisfied with the amount and quality of the information they receive directly from clinicians [27]. Health care professionals may not be aware of patients’ needs and preferences for receiving information [26], and clinicians are known to have poor accuracy in tailoring information to match patient preferences [8,26]. Inadequate communication or withholding information by the health care provider can lead to poor recall or misunderstanding of information [27].

    In addition, written information materials, including leaflets and booklets, often have a reading level that is higher than that of the majority of the population [8,28]. This discrepancy results in information materials that are not readily accessible to some consumers and have the potential to cause unnecessary confusion and anxiety. The most vulnerable consumers—those with low literacy and numeracy—are particularly likely to be affected by the challenges of written information. Varying learning styles and literacy levels may also make it difficult for patients to absorb and engage with written information [29]. Written information materials also depend on the consumer having adequate visual capabilities and therefore are not suitable for visually impaired groups. Currency of information is also important given ongoing developments in the health research literature. Regularly updating the content of printed materials to reflect current best evidence is often not cost-effective, resulting in out-of-date materials remaining in circulation [30]. Written and audio-visual information including videos, DVDs, and CD-ROMs also cannot be tailored to the individual needs of patients and are often very expensive to produce and update as new evidence becomes available [30].


    The Unrealized Potential of Web-Based Information

    Heralded as the future for providing patients with health information in the mid-1990s [30-32], the amount of health information on the Internet has grown exponentially. Surveys of cancer patients and their families have shown that 62-80% have an interest in obtaining information and support via the Internet [33,34]. Patients also frequently report the Internet as an important source of information about diagnosis and treatment [35,36]. The Internet shows great promise in reaching a large proportion of the community; 78% of citizens in the United States have Internet access [37], and new technologies including smartphones and tablet computers have led to a growth in accessing the Internet via mobile devices [38]. Mobile technology improves convenience for users and allows people to connect with Web-based services and information anywhere at any time, which is a feature reported by cancer patients as the main benefit of Web-based information [34]. This provides an ideal platform on which to develop information interventions for improving patient outcomes and health care delivery to a wide audience.

    However, current Web-based information is not without limitations. While the Internet plays an important role in meeting the informational needs of many patients [35], the unmonitored provision of information, coupled with unstructured and unassisted use, has the potential to negatively impact psychological well-being and patient outcomes. There are growing concerns, for example, about the quality of health information available on the Internet [35,39]. Information is often provided without any regulation of credibility of authorship or accuracy of content. This has the potential for dangerous and negative consequences, particularly given that around half of all people who search for health information on the Internet do not discuss the information with their health care provider [40]. The reading level required for many cancer information websites is also far above that of the average population. Friedman and colleagues report that 64% of cancer websites are written at a level of grade 13 or higher [41], which suggests that a large proportion of information may not be accessible by less literate patients. Lack of specificity, complexity, and being too impersonal are also perceived as disadvantages by patients [34].

    These findings highlight the need for more accurate, evidence-based information that is easily understood by consumers, developed in close co-operation with health care professionals, and integrated into clinical care in order to overcome the dangers of misinformation. There is an urgent need to develop a sustainable, systematic, and integrated approach to providing information in a way that addresses health literacy, increases patient involvement in decision making and their care, and operates independently of resource constraints and other barriers that deter the routine delivery of tailored information.


    New-Generation, Integrated, and Tailored Web-Delivered Tools as a Solution for Effective Information Provision

    Overview

    In contrast to the passive dissemination of information via traditional Web-based approaches, new-generation Web-based technology offers an innovative and pragmatic solution to the shortcomings of general Web-based information by providing a platform for interactive information seeking, information sharing, and user-centered design.

    Self-Tailored Content

    Individual patient preferences for information vary. Some patients wish to obtain as much information as possible (monitors), while others prefer to avoid potentially threatening information (blunters) [42]. Tailoring information to match individual preferences improves psychosocial outcomes [12,43] by preventing unnecessary anxiety and increasing recall of information [44]. Web-based information tools have the potential to empower patients to determine when and how often they access information and the type and amount of information they would like, allowing them to become actively involved in their health care. This improves on traditional clinician-delivered information provision where patient preferences may not be considered.

    There is also significant potential for Web-based tools to modify content to account for individual factors such as health literacy and cultural appropriateness. For example, upon login, the user may be asked to complete a brief screening questionnaire, indicating information such as their ethnicity, highest level of education, and preference for detailed versus brief descriptions. These data can be used to tailor the Web-tool content to suit the needs and preferences of the user, for example, by increasing the number of pictures and videos presented or including culturally relevant vocabulary and images. In addition, options can be made available to customize the language (eg, English, Japanese, Spanish) in which the information is presented or increase the size of the text to assist the visually impaired. The interactive nature of Web-based tools allows for real-time customization that is not offered by printed information materials.

    Clinician-Tailored Content

    Algorithms can be used to select the appropriate information to be presented in Web-based information tools based on data entered by the patient or their clinician. However, clinicians should also play a role in tailoring the information presented within a Web-based tool to ensure it is suited to the user’s unique medical circumstances. For example, a Web-based tool can tailor the information presented based on data entered by the clinician regarding the patients’ diagnosis and relevant treatment options. Customizing the content in this way may prevent the patient from feeling unnecessary distress or false hope over viewing treatment information that is not appropriate for their circumstances.

    The involvement of clinicians in tailoring content is important for preventing potential harm caused by misinformation and ensuring the information is relevant to individual patients and their families. Cancer diagnoses can be very complex, and the resulting discussions about treatment options, side effects, and prognosis may require patients to absorb a large amount of complicated information within a short timeframe. The provision of a Web-based information tool that is tailored by the clinician to the unique circumstances of the individual can act as a reliable source of information, supplementary to the doctor-patient consultation. Web-based tools tailored in this way have the capacity to overcome issues such as misinformation [35,37-39], patient comprehension, and information recall [45].

    Multiformat Presentation of Information

    The ability to understand and recall medical information is important for ensuring adherence to recommendations for care [45]. Studies have reported that 40-80% of medical information is forgotten, and about half of the information provided to patients is remembered incorrectly [45]. Levels of anxiety can also negatively impact patient recall of information [45], which is particularly relevant for patients facing a life-threatening illness. A combination of written, spoken, and visual formats has been recommended for improving recall of information [45]. Web-based information tools have the potential to deal with issues surrounding patient memory for medical information and allow information to be presented in multiple formats (eg, diagrams, videos, text) to suit different learning styles [30] and literacy levels [45].

    Connecting People and Information Sharing

    New generation Web technology provides an effective way of connecting patients and families with information and support regardless of their location. For example, social networking sites, online discussion forums, and video-conferencing allow patients and carers, from various and remote locations, to be part of a virtual community, share their experiences and offer support to one another. The availability of this additional support network, which may be otherwise inaccessible, has the potential to improve socialization and reduce feelings of isolation often experienced by patients and their families dealing with a life-threatening illness such as cancer [46]. A survey of hematological cancer survivors found that the second highest ranked item of high-level unmet need was “finding someone to talk to who understands and has been through a similar experience” [47].

    The availability of Web technology encourages the involvement of family and friends in information seeking. Family and friends are able to access information they may wish to know but are reluctant to seek from health care professionals for fear of upsetting the patient, such as information about prognosis, survival, or long-term complications. A Web-based information tool provides a discrete mechanism for accessing this information without the issues of information quality and accuracy often associated with Internet resources. This may allow loved ones to better plan for the future. The instantaneous nature of the Internet also allows for information to be easily shared between patients, their families, and friends. Users are able to access the same information at the same time, send information and links via email, share information and links using social networking sites, and discuss the information using video-conferencing software, such as Skype.

    While Web-based tools allow for information to be shared quickly and easily, there are potential privacy issues that must be considered. In the United States, the Privacy Rule issued under the Health Insurance Portability and Accountability Act of 1996, governs the way in which health information can be shared in order to adequately protect the privacy of individuals while allowing the information flow necessary for the provision of high quality health care [48]. Similarly, a number of countries have legislation covering the recording, storage, and transmission of personal health information. Where Web-based tools require individuals’ health information to be provided by their clinician and stored on a secure server in order to allow for sufficient tailoring of information, it is critical to design the system with a clear understanding of implications of the relevant legislation and in a manner that ensures that adequate privacy and data protections are in place to prevent unauthorized access or sharing of private health information. In addition, where Web-based tools are used to share information with friends and family as described above, it is essential that the tool allows for the patient, or an authorized proxy, to have control of what information is made available to others.

    Inclusion of Decision Support Tools

    The International Patient Decision Aids Standards (IPDAS) Collaboration [49] describes decision aids as evidence-based tools designed to prepare a person for making a decision about their health care options through deliberative exercises such as value weighting [50,51]. Web-based information tools offer a unique opportunity to seamlessly integrate decision aids with the provision of health care information. Web-based tools allow decision aids to be interactive and presented in multiple formats incorporating video, images, and animations, all of which are useful for presenting risk information to patients.

    Integration With Clinical Care Rather Than Passive Patient-Directed Information Seeking

    Web-based information tools may facilitate communication with health care providers. The use of a question prompt sheet within consultations has been shown to be effective in increasing patient participation and reducing unmet information needs [52]. The interactive nature of Web-based tools may enhance the generation and utilization of prompt sheets for patients by tailoring content to correspond with the areas of most interest to the patient. For example, a Web-based information tool has the capacity to generate a question prompt sheet based on the topics of information accessed by the user. The Internet also offers the additional advantage of allowing prompt sheets to be automatically emailed to the health care provider prior to the consultation. This may improve the relevance of the information provided to the patient within the consultation due to increased preparation by the health care provider.

    Distribution, Maintenance, and Feedback

    The ability to track and record how the website is used provides an accurate and detailed process measure for evaluating the intervention. Delivery and maintenance of Web-based information tools have the potential to be highly cost-effective [53] and can be easily updated in a timely manner to reflect current best evidence.


    Framework for Developing an Integrated Web-Based Information Tool

    Overview

    Several reviews of the literature have examined the effectiveness of Web-based interventions for patient education, psychosocial care, and support [54-57]. While these reviews indicate some benefit of Web-based interventions on patient outcomes including knowledge, social support, health behaviors, and psychosocial well-being, findings are mixed and conclusions limited by the methodological weaknesses of the studies examined. Heterogeneity in the methods used to produce and deliver these interventions may contribute to inconclusive findings. There is often a lack of clear, explicit descriptions of the procedures for developing Web-based interventions in the literature, impeding replication and translation of effective interventions. While there are many useful resources available to guide the development of patient information materials in general [58-60], there is a need for a systematic process for the development of patient education and support interventions delivered specifically via the Internet.

    We propose a framework for developing Web-based information tools that draws on our experience of constructing such a tool for hematological cancer patients and their families. The framework is based on principles proposed for the development and evaluation of complex interventions [61,62], which emphasize the importance of using a phased approach, starting with needs assessment, pilot work, and moving on to an exploratory and then definitive evaluation. The framework draws on the Agile methodology of software programming [63], which emphasizes collaboration and iteration throughout the development process. The Agile methodology allows for projects to evolve and be responsive to change, as programmers, researchers, and stakeholders are able to interact to shape the direction of the project through all phases of development.

    Figure 1 illustrates the DoTTI framework for developing a Web-based information tool. The process involves four phases of development: (1) design and development, (2) testing early iterations, (3) testing for effectiveness, and (4) integration and implementation. At each step, stakeholders (including researchers, clinicians, consumers, and programmers) are engaged in consultations to review progress, provide feedback on versions of the Web-based tool, and based on feedback, determine the appropriate next steps in development. Stakeholder participation and iteration have been identified as being essential to the development of effective eHealth technologies [64]. The phases of development are not intended to represent what could be seen as a linear or waterfall approach [65] to the creation of the software underpinning the tool. Rather, the phases represent a staged and concurrently iterative process to the creation of the tool, noting that the development of the informational content and the implementing technology are entwined.

    Figure 1. The DoTTI framework for the iterative and consultative development of a Web-based information tool.
    View this figure

    Phase 1: Design and Development

    Identify the Target Population and Conduct a Needs Assessment to Determine Patients’ Information and Support Needs

    Understanding the needs of the target population is the crucial first step for delivering high quality, patient-centered care [66]. Many existing needs assessment tools assess cancer patients’ needs across a range of domains including physical, psychological, social, financial, and information [66]. While information needs vary, up to 97% of cancer patients report unmet information needs [67]. Common areas where cancer patients report needing additional information include being informed about self-care strategies and the benefits and side effects of treatment [67]. Furthermore, cancer patients tend to report greater unmet needs during the treatment phase [67]. Prior to developing a Web-based information tool, a needs assessment that identifies the informational and support needs of the target population should be undertaken.

    Build on Existing Information Resources and Gain Consensus From Health Care Providers

    Building a partnership with key patient support organizations within the field of interest is advantageous for sharing of resources and expertise. Many support organizations have an established set of information resources available for the target population, which may be used with permission to form the foundation of the Web program content.

    Involving health care professionals in the production, implementation, dissemination, and evaluation of Web-based information tools may increase patients’ use of the tool, adoption by health care organizations, and effectiveness and acceptability [39]. At least one advisory group should be established to provide guidance, advice, and feedback on the content of the intervention and to obtain consensus regarding this information. Expert advisory groups should comprise multidisciplinary health care providers as well as members of key support organizations, so that a wide range of views can be incorporated into the intervention. Building cooperative and respectful relationships with experts is essential for regularly updating the Web-based tool’s content and successful dissemination of the intervention broadly.

    We demonstrate the feasibility of gaining consensus from health professionals through our own experiences during the development of a Web-based information tool. Prior to the consensus meeting, convened in relation to a Web-based tool for hematological patients and their support persons, we allocated sections of content to members for review based on their areas of expertise. Using a rating scale, the quality of the content was scored out of ten in areas such as accuracy, completeness, level of detail, and communication style, allowing identification of areas where improvements were needed. The feedback was then collated and presented at the advisory group meeting. There are various well-established approaches to obtaining consensus, such as the Delphi method, which may be used to determine the content of the Web program [68].

    The consensus meeting provided a valuable opportunity to openly discuss the sections of information that were considered to require significant revisions by reviewers or topics where conflicting feedback was provided. To ensure such meetings are productive and efficient, conflict resolution techniques, such as voting, should be employed when consensus cannot be reached. When consensus cannot be reached and there is high level evidence (for example, clinical practice guidelines or evidence from Cochrane reviews) providing support for one particular intervention over another, the result should be in favor of the evidence. Where such evidence is not available, or the evidence is conflicting, all options should be presented for users. A transparent approach should be employed for communicating information where there is poor or conflicting evidence and no consensus among clinicians. An approach similar to that recommended by Raine and colleagues [69] may be used for increasing the transparency of recommendations where there may be conflicting evidence. Such an approach involves making explicit the reasons for disagreement and the degree of consensus, to assist with decision making.

    Other advantages to involving multidisciplinary health care providers and researchers in a consensus meeting during the development phase of the Web-based tool include the opportunity to (1) discuss the acceptability of the intervention from a health care provider perspective, (2) investigate the probability of health care providers endorsing the information tool and promoting it to patients, and (3) strengthen relationships with clinical colleagues and foster potential future research collaboration.

    Ensure a Well-Constructed and User-Friendly Interface

    To increase the likelihood of use and effectiveness of a Web-based information tool, it is essential that the tool allow the user to extract the desired information as easily as possible. Most literature on effective Web page design emphasizes at least some of Dieter Ram’s design principles [70], namely that good design is innovative, useful, aesthetic, easily understood, unobtrusive, honest, durable, thorough, concerned with the environment, and has “as little design as possible”. Achievement of innovation without being obtrusive or over designed is interesting in the Web context: the current Web 2 browsers support user-provided content and hence user interaction; Web 3 (the semantic Web) adds contextual personalization; and the proposed Web 4 (the symbiotic Web) will be highly intelligent and fully executing. Each new Web form brings with it increased portability, pervasiveness, interactivity, and better support for multimedia (eg, video, audio) [71]. It is important that the Web-based tool uses innovation as appropriate to enhance communication, avoiding unaesthetic, obtrusive glitz by embracing elegant simplicity.

    Another consideration is observation of the patterns followed by users’ eyes when reading Web pages [72]. Research shows that users typically read Web pages in an F-shaped pattern involving a pair of horizontal scans, the first across the top of the content, the second being lower and shorter, followed by a left-oriented vertical scan [72]. This has implications for Web page design as most users will not read all of a page’s content. The first lines of a page should state the most important information in the page, and headings, subheadings, bullet points, and paragraphs should start with words that impart information because these represent the left-most content.

    With regards to the content imparted by the Web-based information tool, its design and implementation must be logical and intuitive to support information seeking for diverse users, including those who are less technologically savvy. The Web pages should be structured in a way that is meaningful to the user and easy to use [73]. Compliance with standard Web design conventions, such as positioning the navigation bar at the top and the organization’s logo in the top-left corner, allows users to easily understand the structure of the website based on their previous experience with other websites [73]. Incorporating various navigation aids, such as a search function, hyperlinks, tabbed menus, and sitemap, improves the ability of users to access desired information and offers navigation flexibility for browsing content [73,74]. Simplicity and consistency in design across all Web pages included in the intervention is essential for effective website navigation [73,74].

    While content must be current, informative, and accurate, the visual design of the Web-based tool is important for capturing the user’s attention. Careful inclusion of graphic features, such as colors, images, and icons, helps to highlight key points and serve to increase the comprehensibility of text-based information [73]. Visual design features such as white space, contrast, and typography are also important considerations for maximizing readability. Ensuring the user interface is attractive and easy to navigate is likely to improve consumers’ use of and satisfaction with the Web-based information tool.

    Phase 2: Testing Early Iterations

    Conduct Alpha and Beta Testing of Early Versions of the Tool

    As early iterations of the Web-based tool are developed, it is essential to follow a test strategy, including alpha and beta testing, to assess the functionality of the tool and ensure it meets the objectives of the project. Alpha testing is often carried out within the project team and typically involves checking for issues such as incorrect or broken links, misspelled words, and problems loading multimedia objects [75]. Following this, beta testing may be undertaken, where the tool is tested by a sample of the intended end-users for any additional defects [75]. At this stage, initial feedback regarding usability may also be sought. During the preliminary testing phase, it may also be beneficial to conduct a heuristic evaluation of the user interface of the Web-based tool leading to improved usability.

    Evaluate the Quality of the Tool Against Established Guidelines

    Checklists and guidelines for assessing the quality of health-related information on the Internet have been developed; however, many are not supported by empirical evidence [76,77]. Several quality criteria for assessing information for consumers of health services include those developed by the King’s Fund, the United Kingdom National Health Service, and Coulter, Entwistle, and Gilbert [77]. These criteria emphasize the need for comprehensive and unbiased information that is presented in a way that is simple and easy to understand and can be integrated into clinical practice. Entwistle and colleagues advocate for additional dimensions of quality including relevance, accuracy, accessibility, comprehensibility, usability, and equity [78]. Other popular assessment tools include the DISCERN instrument [79] and IPDAS [49]. No gold standard quality criterion exists for assessing consumer health information, particularly when the intervention utilizes Internet technology. Evaluating a Web-based information tool against several criteria may provide the best measure of quality and serve to highlight areas where improvement is required.

    Pilot the Web-Based Tool With Patients, Clinicians, and Other Stakeholders

    Obtaining feedback from patients, their families, and other key stakeholders regarding the usability and acceptability of the tool is crucial for maximizing the probability of use and adoption in cancer care settings. Stakeholders are those who have a vested interest in the outcome of the project or initiative and are able to influence the direction it takes [80]. Prior to evaluating the effectiveness of the Web-based information tool, the tool should be pilot-tested with intended users. This procedure allows valuable consumer feedback to be obtained where participants can reflect on their own experiences and offer suggestions about what would have been most helpful to them and their families. The piloting process also provides an opportunity to examine whether the intervention is able to improve health literacy. Feedback from stakeholders could be collected through a variety of sources including surveys, qualitative interviews, or focus groups. Feedback obtained from stakeholders should be evaluated and incorporated prior to assessing the intervention’s effectiveness and included if recommended changes reflect the views of most consumers. Health care providers should again be consulted during this process to ensure the accuracy of amended information.

    The involvement of clinicians throughout the previous development phase is advantageous in helping to ensure the Web-based information tool is acceptable to health care providers and readily integrated with current practice. It is, however, essential to conduct rigorous pilot testing of the acceptability and feasibility of the tool with clinicians who were not involved in Phase 1, given the role played by clinicians in tailoring the content of the tool. Similar techniques may be employed as for pilot testing with patients, such as surveys, qualitative interviews, or focus groups. Measurement of key strokes and eye movements can also be used to ensure that time demands are minimized and ease of use is maximized. This step is crucial for maximizing the probability that the tool will be integrated into clinical practice and adopted by clinicians in Phase 4.

    Phase 3: Testing for Effectiveness

    Test the Effectiveness of the Web-Based Tool in Controlled Studies

    Before the Web-based information tool can be disseminated and adopted, the uptake and effectiveness of the tool for improving patient outcomes should be evaluated with an adequate sample of consumers. Key issues central to the evaluation of effectiveness include: Does the target sample access the intervention? Which patient outcomes are likely to be impacted by the Web-based information tool? and How can effectiveness best be measured? The CONSORT-EHEALTH guidelines should be considered when designing and reporting studies examining the effectiveness of Web-based interventions [81]. While the involvement of consumers in development, as well as rigorous pilot testing conducted in previous phases, is likely to improve the likelihood that the target sample will utilize the Web-based information tool, controlled testing may reveal that uptake is influenced by factors that were not previously considered. It may be necessary to revisit previous phases in order to ascertain the barriers to uptake and revise the tool accordingly. If the intervention is found to be effective in improving patient outcomes, then steps can be taken to broadly disseminate the information tool in a range of clinical settings. Optimal intervention delivery and uptake relies on effective integration into clinical practice. It is essential that health care providers find the information tool acceptable in order to ensure adoption into practice. Partnerships with patient support organizations will help to ensure that the information tool is viewed as credible by providers.

    Phase 4: Integration and Implementation

    Integrate the Tool Into Clinical Practice

    Integration of a Web-based information tool is supported by its capacity to operate independently of health care provider and resource constraints. By automating and standardizing the provision of information, this approach minimizes the burden on physicians, reduces staffing costs, and increases patient convenience. While the burden on clinical staff is minimal, it is essential to provide education and training to ensure the innovative functions of the intervention are utilized to their potential. Educating clinical staff on the advantages of the information tool is likely to improve health care providers’ endorsement and encourage patient uptake. Training clinical staff to tailor the tool’s content to suit patients allows staff to maintain an active role in information provision without the added demands of traditional approaches.

    Ensuring optimal intervention delivery and uptake is a complex issue faced by translational researchers and policy makers globally. The responsibility of educating and training clinical staff in the long term may be best managed by the medical college responsible for delivering care to the particular patient group. The application of evidence from Cochrane reviews [82-84] regarding effective methods of implementation into clinical practice is likely to be of significant benefit; however, this should be considered in conjunction with local protocols for health care delivery and information provision, which may differ between health care settings.

    Monitor and Update the Web-Based Tool as New Evidence Becomes Available

    Coulter, Entwistle, and Gilbert recommend that patient information be based on the best available evidence and be periodically reviewed and updated to reflect advancements [85]. Ensuring information is accurate and up-to-date is essential to informing patients of newly available treatment options and improved services. Partnership with patient support organizations may be extremely beneficial for assisting with this process, as they are likely to be in touch with the needs and preferences of consumers and aware of changing evidence within the field. Developers should also capitalize on the ability to track and record how the Web-based tool is used and obtain real-time feedback from consumers. This valuable data should be used as a means for evaluating and improving the Web-based tool regularly to address the information needs of consumers.

    Conclusions

    New generation Web-based tools that are tailored and integrated into clinical care have the potential to overcome many of the limitations of general Web-based information, thus providing realizable benefits to patients and support persons. The proposed 4-phase DoTTI framework provides a model for a comprehensive approach to the development of Web-based informational tools for patients. The approach is evidence-informed, consumer-centered, flexible, and systematic. Implementation of the framework requires research at key phases including accessibility, acceptability, and effectiveness of each tool.

    Acknowledgments

    This research is supported by a New South Wales Cancer Institute Translational Health Services Research Grant. This manuscript was also supported by a Strategic Research Partnership Grant from Cancer Council NSW to the Newcastle Cancer Control Collaborative. Dr Bryant is supported by an Australian Research Council Post-Doctoral Industry Fellowship. Dr Tzelepis was supported by a Leukaemia Foundation of Australia and Cure Cancer Australia Foundation Post-Doctoral Research Fellowship.

    Authors' Contributions

    All authors were involved in the development of the Web tool for hematological cancer patients that informs the framework. RS and JB played a major role in concept development and manuscript preparation. RSF contributed to concept development and critical review of the manuscript. FT, CP, and WS assisted with preparation and critical review of the manuscript. FH provided expert knowledge of the software programming aspects of the framework and assisted with manuscript preparation.

    Conflicts of Interest

    None declared.

    References

    1. Ferlay J, Shin HR, Bray F, Forman D, Mathers C, Parkin DM. Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008. Int J Cancer 2010 Dec 15;127(12):2893-2917. [CrossRef] [Medline]
    2. Payne S, Jarrett N, Jeffs D. The impact of travel on cancer patients' experiences of treatment: a literature review. Eur J Cancer Care (Engl) 2000 Dec;9(4):197-203. [Medline]
    3. Zabora J, BrintzenhofeSzoc K, Curbow B, Hooker C, Piantadosi S. The prevalence of psychological distress by cancer site. Psychooncology 2001;10(1):19-28. [Medline]
    4. Delgado-Guay M, Parsons HA, Li Z, Palmer JL, Bruera E. Symptom distress in advanced cancer patients with anxiety and depression in the palliative care setting. Support Care Cancer 2009 May;17(5):573-579. [CrossRef] [Medline]
    5. Ayres A, Hoon PW, Franzoni JB, Matheny KB, Cotanch PH, Takayanagi S. Influence of mood and adjustment to cancer on compliance with chemotherapy among breast cancer patients. J Psychosom Res 1994 Jul;38(5):393-402. [Medline]
    6. Simon G, Ormel J, VonKorff M, Barlow W. Health care costs associated with depressive and anxiety disorders in primary care. Am J Psychiatry 1995 Mar;152(3):352-357. [Medline]
    7. Husson O, Mols F, van de Poll-Franse LV. The relation between information provision and health-related quality of life, anxiety and depression among cancer survivors: a systematic review. Ann Oncol 2011 Apr;22(4):761-772 [FREE Full text] [CrossRef] [Medline]
    8. Mills ME, Sullivan K. The importance of information giving for patients newly diagnosed with cancer: a review of the literature. J Clin Nurs 1999 Nov;8(6):631-642. [Medline]
    9. Mccaffery KJ, Shepherd HL, Trevena L, Juraskova I, Barratt A, Butow PN, Sydney Health Decision Group. Shared decision-making in Australia. Z Arztl Fortbild Qualitatssich 2007;101(4):205-211. [Medline]
    10. Goldberg RJ. Disclosure of information to adult cancer patients: issues and update. J Clin Oncol 1984 Aug;2(8):948-955. [Medline]
    11. Gogos AJ, Clark RB, Bismark MM, Gruen RL, Studdert DM. When informed consent goes poorly: a descriptive study of medical negligence claims and patient complaints. Med J Aust 2011 Sep 19;195(6):340-344. [Medline]
    12. Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, D.C: National Academy Press; 2001.
    13. American Hospital Association. The Patient Care Partnership: Understanding Expectations, Rights and Responsibilities.: American Hospital Association; 2003.   URL: http://www.aha.org/content/00-10/pcp_english_030730.pdf [accessed 2013-11-12] [WebCite Cache]
    14. Australian Commission on Safety and Quality in Health Care. Australian Charter of Healthcare Rights. Sydney: Australian Commission on Safety and Quality in Health Care; 2008.   URL: http://www.safetyandquality.gov.au/wp-content/uploads/2012/01/Charter-PDf.pdf [accessed 2013-11-12] [WebCite Cache]
    15. Council on Ethical and Judicial Affairs. Code of Medical Ethics: Current Opinions with Annotations 2012-2013 edition. Chicago: American Medical Association; 2013.
    16. Maizes V, Rakel D, Niemiec C. Integrative medicine and patient-centered care. Explore (NY) 2009;5(5):277-289. [CrossRef] [Medline]
    17. Boote J, Telford R, Cooper C. Consumer involvement in health research: a review and research agenda. Health Policy 2002 Aug;61(2):213-236. [Medline]
    18. Macmillan Cancer Support.   URL: http://www.macmillan.org.uk/Home.aspx [accessed 2013-11-12] [WebCite Cache]
    19. Cancer Voices Australia.   URL: http://www.cancervoicesaustralia.org/ [accessed 2013-11-12] [WebCite Cache]
    20. LiveStrong.   URL: http://www.livestrong.org/ [accessed 2013-11-12] [WebCite Cache]
    21. Bastian H. Speaking up for ourselves. The evolution of consumer advocacy in health care. Int J Technol Assess Health Care 1998;14(1):3-23. [Medline]
    22. Chewning B, Bylund CL, Shah B, Arora NK, Gueguen JA, Makoul G. Patient preferences for shared decisions: a systematic review. Patient Educ Couns 2012 Jan;86(1):9-18. [CrossRef] [Medline]
    23. Davis TC, Williams MV, Marin E, Parker RM, Glass J. Health literacy and cancer communication. CA Cancer J Clin 2002;52(3):134-149. [Medline]
    24. Heckinger EA, Bennet CL, Davis T, Wolf MS. Health literacy and cancer care: Do patients really understand? Journal of Clinical Oncology (2004 ASCO Annual Meeting Proceedings) 2004;22(14S):6045.
    25. Degner LF, Kristjanson LJ, Bowman D, Sloan JA, Carriere KC, O'Neil J, et al. Information needs and decisional preferences in women with breast cancer. JAMA 1997 May 14;277(18):1485-1492. [Medline]
    26. Elkin EB, Kim SH, Casper ES, Kissane DW, Schrag D. Desire for information and involvement in treatment decisions: elderly cancer patients' preferences and their physicians' perceptions. J Clin Oncol 2007 Nov 20;25(33):5275-5280. [CrossRef] [Medline]
    27. Jefford M, Tattersall MH. Informing and involving cancer patients in their own care. Lancet Oncol 2002 Oct;3(10):629-637. [Medline]
    28. Butow P, Brindle E, McConnell D, Boakes R, Tattersall M. Information booklets about cancer: factors influencing patient satisfaction and utilization. Patient Educ Couns 1998 Feb;33(2):129-141. [Medline]
    29. von Wagner C, Semmler C, Good A, Wardle J. Health literacy and self-efficacy for participating in colorectal cancer screening: The role of information processing. Patient Educ Couns 2009 Jun;75(3):352-357. [CrossRef] [Medline]
    30. Richards B, Colman AW, Hollingsworth RA. The current and future role of the Internet in patient education. Int J Med Inform 1998 Jun;50(1-3):279-285. [Medline]
    31. Doyle DJ, Ruskin KJ, Engel TP. The Internet and medicine: past, present, and future. Yale J Biol Med 1996;69(5):429-437 [FREE Full text] [Medline]
    32. Wallace S. Health information in the new millennium and beyond: the role of computers and the Internet. Health Education 1997;97(3):88-95. [CrossRef]
    33. Monnier J, Laken M, Carter CL. Patient and Caregiver Interest in Internet-Based Cancer Services. Cancer Pract 2002 Nov;10(6):305-310. [CrossRef]
    34. Paul CL, Carey ML, Hall AE, Lynagh MC, Sanson-Fisher RW, Henskens FA. Improving access to information and support for patients with less common cancers: hematologic cancer patients' views about Web-based approaches. J Med Internet Res 2011;13(4):e112 [FREE Full text] [CrossRef] [Medline]
    35. Mathieu E. The Internet and medical decision making: can it replace the role of health care providers? Med Decis Making 2010;30(5 Suppl):14S-16S. [CrossRef] [Medline]
    36. Peterson MW, Fretz PC. Patient use of the internet for information in a lung cancer clinic. Chest 2003 Feb;123(2):452-457. [Medline]
    37. The World Bank. World Development Indicators. 2012. Internet users (per 100 people)   URL: http://data.worldbank.org/indicator/IT.NET.USER.P2 [accessed 2013-11-12] [WebCite Cache]
    38. The Nielsen Company. The Australian online consumer landscape. Macquarie Park, NSW: Nielsen; 2012.   URL: http:/​/www.​mobilemediamarketing.com.au/​wp-content/​uploads/​2012/​04/​Australian_Online_Landscape_Report_2012_-March.​pdf [accessed 2013-11-12] [WebCite Cache]
    39. Benigeri M. Shortcomings of health information on the Internet. Health Promotion International 2003 Dec 01;18(4):381-386. [CrossRef]
    40. Houston TK, Allison JJ. Users of Internet health information: differences by health status. J Med Internet Res 2002;4(2):E7 [FREE Full text] [CrossRef] [Medline]
    41. Friedman DB, Hoffman-Goetz L, Arocha JF. Readability of cancer information on the internet. J Cancer Educ 2004;19(2):117-122. [CrossRef] [Medline]
    42. Miller SM. Monitoring versus blunting styles of coping with cancer influence the information patients want and need about their disease. Implications for cancer screening and management. Cancer 1995 Jul 15;76(2):167-177. [Medline]
    43. National Breast Cancer and National Cancer Control Initiative. Clinical practice guidelines for the psychosocial care of adults with cancer. Camperdown, NSW: National Breast Cancer Centre; 2003.   URL: http://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/cp90.pdf [accessed 2013-11-12] [WebCite Cache]
    44. McPherson CJ, Higginson IJ, Hearn J. Effective methods of giving information in cancer: a systematic literature review of randomized controlled trials. J Public Health Med 2001 Sep;23(3):227-234 [FREE Full text] [Medline]
    45. Kessels RP. Patients' memory for medical information. J R Soc Med 2003 May;96(5):219-222 [FREE Full text] [Medline]
    46. Ziebland S, Chapple A, Dumelow C, Evans J, Prinjha S, Rozmovits L. How the internet affects patients' experience of cancer: a qualitative study. BMJ 2004 Mar 6;328(7439):564 [FREE Full text] [CrossRef] [Medline]
    47. Hall A, Campbell HS, Sanson-Fisher R, Lynagh M, D'Este C, Burkhalter R, et al. Unmet needs of Australian and Canadian haematological cancer survivors: a cross-sectional international comparative study. Psychooncology 2013 Sep;22(9):2032-2038. [CrossRef] [Medline]
    48. Office for Civil Rights. Summary of the HIPPA Privacy Rule. Washington, D.C: United States Department of Health and Human Services; 2003.   URL: http://www.hhs.gov/ocr/privacy/hipaa/understanding/summary/privacysummary.pdf [accessed 2013-11-12] [WebCite Cache]
    49. International Patient Decision Aid Standards Collaboration. Criteria for judging the quality of patient decision aids. Ottawa: International Patient Decision Aid Standards Collaboration; 2006.   URL: http://ipdas.ohri.ca/IPDAS_checklist.pdf [accessed 2013-11-12] [WebCite Cache]
    50. Clinton-McHarg T, Paul C, Sanson-Fisher R, D'Este C, Williamson A. Determining research priorities for young people with haematological cancer: a value-weighting approach. Eur J Cancer 2010 Dec;46(18):3263-3270. [CrossRef] [Medline]
    51. Paul CL, Sanson-Fisher R, Douglas HE, Clinton-McHarg T, Williamson A, Barker D. Cutting the research pie: a value-weighting approach to explore perceptions about psychosocial research priorities for adults with haematological cancers. Eur J Cancer Care (Engl) 2011 May;20(3):345-353 [FREE Full text] [CrossRef] [Medline]
    52. Clayton JM, Butow PN, Tattersall MH, Devine RJ, Simpson JM, Aggarwal G, et al. Randomized controlled trial of a prompt list to help advanced cancer patients and their caregivers to ask questions about prognosis and end-of-life care. J Clin Oncol 2007 Feb 20;25(6):715-723. [CrossRef] [Medline]
    53. Murray E. Web-Based Interventions for Behavior Change and Self-Management: Potential, Pitfalls, and Progress. Med 2.0 2012 Aug 14;1(2):e3 [FREE Full text] [CrossRef]
    54. Murray E, Burns J, See TS, Lai R, Nazareth I. Interactive Health Communication Applications for people with chronic disease. Cochrane Database Syst Rev 2005(4):CD004274. [CrossRef] [Medline]
    55. Nguyen HQ, Carrieri-Kohlman V, Rankin SH, Slaughter R, Stulbarg MS. Internet-based patient education and support interventions: a review of evaluation studies and directions for future research. Comput Biol Med 2004 Mar;34(2):95-112. [CrossRef] [Medline]
    56. Paul CL, Carey ML, Sanson-Fisher RW, Houlcroft LE, Turon HE. The impact of web-based approaches on psychosocial health in chronic physical and mental health conditions. Health Educ Res 2013 Jun;28(3):450-471 [FREE Full text] [CrossRef] [Medline]
    57. Ryhänen AM, Siekkinen M, Rankinen S, Korvenranta H, Leino-Kilpi H. The effects of Internet or interactive computer-based patient education in the field of breast cancer: a systematic literature review. Patient Educ Couns 2010 Apr;79(1):5-13. [CrossRef] [Medline]
    58. AMC Cancer Research Centre, Centres for Disease Control and Prevention. Beyond the brochure: Alternative approaches to effective health communication. Denver, Colorado: AMC Cancer Research Centre; 1994.   URL: http://www.cdc.gov/cancer/nbccedp/pdf/amcbeyon.pdf [accessed 2013-11-12] [WebCite Cache]
    59. Duman M. Producing Patient Information: How to Research, Develop and Produce Effective Information Resources. London: King's Fund; 2003.
    60. National Cancer Institute (NIH). Clear & Simple: Developing Effective Print Materials for Low-Literate Readers. Bethesda, Maryland: National Cancer Institute (NIH); 2003.   URL: http://www.cancer.gov/cancertopics/cancerlibrary/clear-and-simple/page1 [accessed 2013-11-12] [WebCite Cache]
    61. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655 [FREE Full text] [Medline]
    62. McKenzie JF, Neiger BL, Smeltzer JL. Planning, implementing, and evaluating health promotion programs: a primer. San Francisco: Pearson/Benjamin Cummings; 2005.
    63. Shore JH, Warden S. The Art of Agile Development. Sebastopol, CA: O'Reilly Media, Inc; 2007.
    64. van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011;13(4):e111 [FREE Full text] [CrossRef] [Medline]
    65. Royce W. Managing the development of large software systems. Proceedings of WESCON, IEEE CS Press 1970;26:328-339.
    66. Richardson A, Medina J, Brown V, Sitzia J. Patients' needs assessment in cancer care: a review of assessment tools. Support Care Cancer 2007 Oct;15(10):1125-1144. [CrossRef] [Medline]
    67. Harrison JD, Young JM, Price MA, Butow PN, Solomon MJ. What are the unmet supportive care needs of people with cancer? A systematic review. Support Care Cancer 2009 Aug;17(8):1117-1128. [CrossRef] [Medline]
    68. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995 Aug 5;311(7001):376-380 [FREE Full text] [Medline]
    69. Raine R, Sanderson C, Black N. Developing clinical guidelines: a challenge to current methods. BMJ 2005 Sep 17;331(7517):631-633 [FREE Full text] [CrossRef] [Medline]
    70. Lovell S. As Little Design As Possible: The Work of Dieter Rams. London: UK Phaidon Press; 2011.
    71. Wallis M, Henskens FA, Hannaford MR. Classification and Modelling of Web Technologies. 7th International Conference on Web Information Systems & Technologies (WEBIST-2011): INSTICC/ACM-SIGMIS 2011:151-156.
    72. Nielsen J, Pernice K. Eyetracking Web Usability (Voices That Matter). Berkeley, CA: New Riders Press; 2009.
    73. Tan GW, Wei KK. An empirical study of Web browsing behaviour: Towards an effective Website design. Electronic Commerce Research and Applications 2006 Dec;5(4):261-271. [CrossRef]
    74. Lynch PJ, Horton S. Web style guide: basic design principles for creating Web sites. New Haven, Conn: Yale University Press; 2008.
    75. Kaner C, Nguyen HT, Falk JL. Testing computer software. New York: Wiley Computer Publishing; 1999.
    76. Daraz L, MacDermid JC, Wilkins S, Shaw L. Tools to evaluate the quality of web health information: A structured review of content and usability. The International Journal of Technology, Knowledge & Society 2009;5(3):126-141.
    77. National Health and Medical Research Council. How to prepare and present evidence-based information for consumers of health services: A literature review. Canberra: National Health and Medical Research Council; 1999.   URL: http://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/cp72.pdf [accessed 2013-11-12] [WebCite Cache]
    78. Entwistle VA, Sheldon TA, Sowden AJ, Watt IS. Supporting consumer involvement in decision making: what constitutes quality in consumer health information? Int J Qual Health Care 1996 Oct;8(5):425-437. [Medline]
    79. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999 Feb;53(2):105-111 [FREE Full text] [Medline]
    80. Nielsen CP, Lauritsen SW, Kristensen FB, Bistrup ML, Cecchetti A, Turk E, European network for Health Technology Assessment Work Package 6 Partners. Involving stakeholders and developing a policy for stakeholder involvement in the European network for health technology assessment, EUnetHTA. Int J Technol Assess Health Care 2009 Dec;25 Suppl 2:84-91. [CrossRef] [Medline]
    81. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011;13(4):e126 [FREE Full text] [CrossRef] [Medline]
    82. Flodgren G, Eccles MP, Shepperd S, Scott A, Parmelli E, Beyer FR. An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes. Cochrane Database Syst Rev 2011(7):CD009255. [CrossRef] [Medline]
    83. Forsetlund L, Bjørndal A, Rashidian A, Jamtvedt G, O'Brien MA, Wolf F, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2009(2):CD003030. [CrossRef] [Medline]
    84. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;6:CD000259. [CrossRef] [Medline]
    85. Coulter A, Entwistle V, Gilbert D. Sharing decisions with patients: is the information good enough? BMJ 1999 Jan 30;318(7179):318-322 [FREE Full text] [Medline]


    Abbreviations

    CD-ROM: compact disc, read-only-memory
    CONSORT: Consolidated Standards of Reporting Trials
    DoTTI: Design and develOpment, Testing early iterations, Testing for Effectiveness, Integration and implementation
    DVD: digital versatile disc
    IPDAS: International Patient Decision Aids Standards


    Edited by G Eysenbach; submitted 24.07.13; peer-reviewed by C Vinson, S Aranda; comments to author 19.09.13; revised version received 14.11.13; accepted 20.01.14; published 14.03.14

    ©Rochelle Smits, Jamie Bryant, Rob Sanson-Fisher, Flora Tzelepis, Frans Henskens, Christine Paul, William Stevenson. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 14.03.2014.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.