Advertisement: Preregister now for the Medicine 2.0 Congress
A Multimedia Interactive Education System for Prostate Cancer Patients: Development and Preliminary Evaluation
Michael A Diefenbach1, PhD; Brian P Butz2, PhD
1Fox Chase Cancer Center, Division of Population Science, Philadelphia PA, USA
2Temple University, College of Electrical Engineering, Philadelphia PA, USA
Fox Chase Cancer Center
Division of Population Science
333 Cottman Ave
Philadelphia PA 19111-2497
Phone: +1 215 728 5644
Fax: +1 215 214 2178
Background: A cancer diagnosis is highly distressing. Yet, to make informed treatment choices patients have to learn complicated disease and treatment information that is often fraught with medical and statistical terminology. Thus, patients need accurate and easy-to-understand information.
Objective: To introduce the development and preliminary evaluation through focus groups of a novel highly-interactive multimedia-education software program for patients diagnosed with localized prostate cancer.
Methods: The prostate interactive education system uses the metaphor of rooms in a virtual health center (ie, reception area, a library, physician offices, group meeting room) to organize information. Text information contained in the library is tailored to a person's information-seeking preference (ie, high versus low information seeker). We conducted a preliminary evaluation through 5 separate focus groups with prostate cancer survivors (N = 18) and their spouses (N = 15).
Results: Focus group results point to the timeliness and high acceptability of the software among the target audience. Results also underscore the importance of a guide or tutor who assists in navigating the program and who responds to queries to facilitate information retrieval.
Conclusions: Focus groups have established the validity of our approach and point to new directions to further enhance the user interface.
(J Med Internet Res 2004;6(1):e3)
Multimedia software; multimedia; software; prostate cancer; patient education; treatment decision making; treatment; decision making
There is hardly a more-distressing health event in a person's life than receiving a cancer diagnosis. Such a diagnosis often has been compared with a "blow to the head," or the feeling that the "world has changed forever." Often, intense feelings of anxiety and fear about oneself and the future of family members are triggered by a cancer diagnosis. Yet, to make informed decisions about their health care during this time of high distress, patients and their family members need to process complicated medical information that is fraught with medical and probabilistic terms.
This is particularly true for men diagnosed with early-stage prostate cancer (ie, tumors confined to the prostate without involvement of regional lymph nodes or distant metastasis). Although prostate cancer is a highly-treatable disease with an average survival rate of 97% over 5 years , there is no one standard treatment that is appropriate for every patient [2,3]. Patients usually have the choice between surgical removal of the prostate (ie, prostatectomy), external beam radiation, or implantation of radioactive seeds. Watchful waiting, or close monitoring of disease progression through frequent prostate specific antigen (PSA) levels and digital rectal examinations (DREs), is sometimes an option. In addition, patients' treatment decision making is further complicated by the tendency of physicians to recommend treatment options that represent their medical specialty . Thus, newly-diagnosed prostate cancer patients not only have to cope with the emotional impact of the diagnosis, but also have to make sense of complicated medical-treatment information, resolve often-times diverging medical opinions about how to best treat the disease, as well as to anticipate potential unpleasant treatment-related side-effects (ie, incontinence and erectile dysfunction ).
In recognition of these unique patient-information needs the primary aim of this paper is to describe the development and preliminary evaluation of a novel and innovative computer-based multimedia prostate interactive education system (PIES) and treatment-decision tool. We will begin with a brief overview of the most-common treatment options for prostate cancer and their likely consequences for quality of life. Second, we will describe how patients' information-seeking behavior has changed with the development of personal computers and the Internet. Third, we will place the development of PIES in the larger context of computer-aided learning. Fourth, we provide a detailed description of PIES and a report of its preliminary evaluation through focus groups. The paper closes with recommendations for further developments of computer-based education systems and decision tools.
Since the invention of the electronic computer there have been 2 divergent opinions as to how this device should be used. One group saw the computer's principal use as a number manipulator, an extensive, ultrafast, and accurate calculator. Another group envisioned the computer as a symbol manipulator that might be taught to use logic and make decisions in a human fashion. The symbol-manipulation group eventually founded the discipline called artificial intelligence (AI). As time passed, artificial intelligence spawned many subdisciplines such as robotics, natural language recognition, and expert systems. The driving force behind expert-system development was the attempt to program a computer so it could "think" and make human-like decisions. A generally-accepted definition of an expert system is one that performs at or near the level of a human expert in a particular field of endeavor .
The first large expert system to perform at the level of a human expert was developed for use in the health care field . This system, called MYCIN, was developed at Stanford University in the mid 1970s. The MYCIN program was designed to provide physicians with advice about bacteremia and meningitis; it emulated the expertise from a physician specializing in these infections. MYCIN asks the physician, who is requesting its advice, several questions. Additionally, it requires personal data about the patient and will ask for results of laboratory tests when it needs this information to aid in its analysis process. After analyzing patient information and data, MYCIN makes a diagnosis as well as a drug-therapy recommendation. MYCIN also has the ability to explain how it arrived at its diagnosis .
The architecture of the MYCIN program consists of 2 major, distinct elements: (1) a knowledge base and (2) an inference engine. The knowledge base contains the facts and heuristics necessary to derive medical analyses of bacteremia and meningitis infections. The inference engine contains the inference mechanisms and control strategies employed by MYCIN to derive the diagnosis and treatment recommendation. It soon became clear that this architecture was not just specific to MYCIN but could be used as a template or "shell" for encoding and applying expertise in any field.
Computer-Aided Learning for Educational Purposes
One of the most interesting applications of expert system technology is for education. One goal among some artificial-intelligence researchers has been to develop a computerized tutor that performs at the level of an excellent human tutor. The resulting computer program that achieves this objective would be called an intelligent tutoring system (ITS). The effort to create truly-intelligent computer-based tutors has been underway for many decades . Ultimately intelligent tutoring systems attempt to simulate the behavior of an intelligent human tutor in addition to acting as a domain expert. The characteristics of an intelligent tutoring system include the ability [10,11] to:
- teach a given subject
- detect student errors
- analyze where and how the student made an error
- correct flaws in the student's logic
- clear up any misgivings or misunderstandings the student may have about the material.
Although the development of expert systems for training has been moderately successful and other kinds of expert systems, such as income-tax preparation assistants, have been extremely successful commercially, intelligent-tutoring-system development did not advance significantly for several years after its initial efforts. It is within the last decade that prospects for intelligent tutoring systems have shown promise. Advancements in hardware, operating systems, and interactive-multimedia development tools have created a software environment that makes real intelligent tutoring systems a possibility [12- 14].
Interactive multimedia software is already playing a key and unique role in the educational process. It is used as a stand-alone educational module that is intended to enable an interested individual to learn about a particular topic or subject . It has been used as a supplement to classroom presentations [16,17] and laboratories [18,19], and it is used as a dynamic textbook . Recently, interactive multimedia have been integrated with expert-system technology, producing highly-interactive intelligent tutoring systems [21- 25]. These intelligent tutoring systems "learn" about the individual student and can tailor the material to meet the needs and the learning styles appropriate to the individual student.
Computer-Aided Learning for Health Care
One of the first successful large-scale expert systems was developed for diagnosis and treatment of certain infections . The principles of this approach were subsequently applied to include the application of artificial intelligence techniques to medical diagnosis (for example, [27- 32]). Artificial intelligence research within medicine is very active today although attempts to develop physician-like diagnostic systems have waned. Instead, these systems are now geared to assist students and physicians to develop and hone diagnostic skills.
Because enthusiasm for medical diagnosis systems that would "assist" physicians and might even dispute their diagnoses was tepid, the development of systems designed for practitioner education has been advanced. Consequently, considerable effort has been made to produce software systems that could simulate medical situations and include a computerized mentor that would assist learners in their decision-making process. The rapid development of computer hardware has eliminated many of the early problems encountered in developing medical-training expert systems. Today's graphical user interfaces coupled with developments in various video and audio digital circuit boards have encouraged the development and use of multimedia software. Multimedia is usually defined as "the use of a computer to present and combine text, graphics, audio and video with links and tools that allow the user to navigate, interact, create, and communicate" . Multimedia programs have been written to help educate medical practitioners in almost every aspect of health delivery. Systems exist to train and educate individuals in the principles of: laparoscopic procedures and epilepsy diagnosis and treatment , ureterorenoscopy , hysteroscopy , dermatology , diabetes management , communicable and rare diseases , red cell antibody identification , cervical cytology , psychotherapy , and many other health-related areas.
Besides educating practitioners, progress has been made recently in developing interactive multimedia software products for patient use and education. Agre et al, reporting on a project that develops a CD-ROM for cancer-related patient education  notes that CD-ROMs are more valuable for learning than booklets or videotapes. The study finds that CD-ROM technology allows for greater depth in content and has the ability to satisfy a broad range of educational needs. Guendelman et al  have used a computer-based self-management program called Health Buddy to enable children to assess and monitor their asthma symptoms and quality of life as well as to transmit this information to health care providers. Using a randomized control trial, 66 individuals were placed in the intervention group while 68 were in the control group. The study found that compared with an asthma diary, monitoring asthma symptoms and functional status with the Health Buddy increases self-management skills and improves asthma outcomes. Another recently-developed CD-ROM focused on the education of children with leukemia (ages 4-11) and their families about the disease and its treatment .
Multimedia has also been used, with some success, for educational intervention for low-education, low-income Latinas . At the conclusion of the study, it was found that about 40% of the women that had scheduled or received mammography had attributed their decision to the intervention. Interactive multimedia has been used to deliver information: about self-medication , for patient colonoscopy education , for genital herpes education , for personal care of diabetes , for skin cancer prevention , for nutrition screening and counseling , for hypertensive patients , and for numerous other diseases or treatments.
Gustafson et al have developed a computer-based health information and support system [52,53]. This Comprehensive Health Enhancement Support System (CHESS) is a Web-based support system that includes modular programs on breast cancer, AIDS/HIV infection, sexual assault, alcoholism, and academic crisis. For example, the breast cancer module provides disease information, a treatment decision aid, an opportunity to contact health care providers via e-mail, testimonials from patients, and a patient forum to exchange information and to solicit social support. Evaluation studies have confirmed that the CHESS system is user-friendly and well accepted by patients of diverse demographic background . The use of the CHESS system has also been linked to improvements of quality of life, and increased participation in health care among HIV-positive patients . Despite its sophistication, the CHESS system lacks the capacity to present information in a targeted and tailored manner based on specific patient characteristics.
Computer-Aided Tailored Patient Education
A decade ago researchers  noted that computer software has the potential to elicit information from patients and to use this information to tailor patient education. Because the literature is not always consistent in the use of the terms targeted and tailored with respect to health communications, a recent article has suggested clarification and standardization in terminology . The authors suggested that health communications be labeled as targeted if they are directed towards a particular subgroup of the population, and to use the term tailored if the information is designed to address individual characteristics of a person. Both targeting and tailoring depend on the assessment of group or individual characteristics that have been derived from prior data collections .
The goal of targeting and tailoring health communications is to increase the personal relevance of the message to the individual. Research has demonstrated that messages that are relevant to the person are better understood and better remembered [56,57]. In addition, personally-relevant messages are also more likely to lead to behavior change . Thus, it is generally expected that an individual will react to tailored messages with increased attention, improved understanding, and an increased tendency for behavior change [54,59].
Overview of the Prostate Interactive Education System (PIES)
The prostate interactive education system (PIES) is an interactive multimedia expert system that is being developed to help patients who have been diagnosed with early-stage prostate cancer. PIES provides the patient with a multitude of treatment information and encourages patients to obtain the kind of information that they desire about the disease. PIES is envisioned as an important step in the treatment-decision process. The software will enable a patient to follow a logical decision process and obtain the information he needs at his current state of decision-making. The software will prepare a factual report for a psychologist who will meet with the patient following his interaction with the software. The temporal placement of the software is shown in Figure 1. Once the patient is diagnosed with prostate cancer, he is offered the PIES software package to assist him in making a treatment decision. PIES provides additional information and does not preclude a second opinion, as shown in Figure 1. In fact, the patient may receive a second opinion before using PIES.
[view this figure]
|Figure 1. Role of prostate interactive education system (PIES) in the treatment decision process|
The purpose of the PIES software is threefold:
- It allows a patient to obtain the information he wants in a quiet environment. The patient interacts at his own speed. Questions can be asked in any order. Multiple physicians are consulted without any implied obligation on the patient's part. A patient may consult with a surgeon and a radiologist. Consequently, the patient can gather information in a nonthreatening and unpressured environment.
- It allows the patient to access the experiences of other patients. An extensive library of interactive videos of others' experiences will be available to the patient. The patient can choose to hear from videotaped actual patients who are in the same age group, of the same race, and who had a particular treatment.
- It provides an expert system that will be operating in the background. The expert system will analyze what the patient is doing and determine if the patient is getting enough information to make an informed decision. The expert system will not make any decision for the patient. It will only follow and evaluate the completeness of the process. If the expert system becomes concerned that the patient is not being thorough in his examination of alternatives, it will make suggestions to view information that has not been accessed and about where the patient might find it. The patient can follow or ignore the expert system's suggestions.
PIES is not a diagnosis tool. Nor does it give advice in the form of "this is what you should do." Instead, it allows patients to ask questions and get information by interacting with a multimedia computer program. The multimedia program allows a patient to visit a library to get up-to-date information about prostate cancer and its treatment. The patient will be able to interact with virtual physicians who can answer specific questions that the patient may have. Various physicians (eg, surgeons, radiologists) can be queried by the patient. The physicians "consult" with the patient through digital video sequences as well as through interactive multimedia question-and-answer sessions. The patient will also be able to "visit" a virtual support room where he will have the chance to listen to other patients talking about the disease and how they determined their own course of treatment. Support group members also discuss dealing with the side effects of their treatment. The patient will also be able to consult with a virtual sex therapist who will discuss methods and assistive technologies that will enable them to continue sexual activities. Finally, each patient will be provided with a decision aid that will assist him in treatment decision making.
An overview of the PIES architecture is shown in Figure 2. The patient interacts with the CD-ROM based system on a personal computer. The system is CD-ROM based but could be made available over the Internet. A CD-ROM delivery system was chosen so that those patients without a high-speed connection to the Internet would not experience delays caused by downloading videos. The information is developed and presented using Macromedia's Authorware . Authorware is able to communicate with other software and can transmit information in real time or near real time. Authorware will allow the patient to have access to other programs such as a notebook to record information, and various applicable medical-applications software.
[view this figure]
|Figure 2. The PIES architecture|
The patient model, described later, is a computer characterization of the patient; it is changed dynamically by the expert system. In turn, the expert system, also described later, analyzes how the patient interacts with the software and reports this in the results folder. The results folder contains information that the expert system will use to generate a report to the psychologist.
The paradigm used in PIES is that of a patient entering a health center. When the patient enters PIES, he is greeted by an information specialist. The information specialist welcomes the patient and shows him around the PIES Health Center. The Health Center consists of a reception area, a library, and a group meeting room on the first floor, and physician offices on the second and third floors, connected through an elevator. Each room is interactive and the patient is shown how to use the room's facilities when he first chooses to enter the room. After showing the patient the Health Center layout, the information specialist asks the patient to complete a questionnaire. The questionnaire requests data that the expert system will need to tailor the Health Center for his needs.
After completing the questionnaire, the patient is free to visit any room of his choosing.
Elements of PIES
If the patient would like information, he is referred to the library. To get to the library, or any other room, the patient would click with the mouse on the door that leads to the desired area. The library is a highly-interactive area (see Figure 3) in which the patient may obtain, and interact with, educational material and other information. The library consists of a wall of shelves with many books to choose from. It also has a TV through which videos can be accessed. The books are sorted in alphabetical order; by "rolling over" a book with a mouse, its title is revealed. For example, a book entitled Brachytherapy (see Figure 4) contains information about radioactive-seed implant treatment. A chapter gives an overview of brachytherapy; another chapter focuses on side effects, while another one describes the rationale behind a particular treatment regimen. Other books available contain information about psychosocial functioning, such as how to deal with impotence and incontinence, the use of alternative medicine, clinical trials, and the impact of prostate cancer on the family. The video section contains short videos (up to 5 minutes) that show facilities (eg, a surgical suite) and describe specific treatments. In sum, the library is a place that has a comprehensive collection of disease- and treatment-related materials that not only address the momentary concerns of the patient, but also assist in the preparation for future prostate-cancer and treatment-related consequences.
[view this figure]
|Figure 3. The PIES library|
Sources of Information
The identification and development of the information contained in PIES is a multi-step process. A starting point for compiling information is a review of current information on prostate cancer treatment (eg, the PDQ ), that is approved by the National Cancer Institute  and the Cancer Information Service . Regular Medline  searches supplement this information and ensure that it stays current. Second, the research team summarizes this information and identifies and develops appropriate visual materials. Third, medical expert consultants vet all materials for accuracy. Fourth, a health educator and a cancer-information specialist adopt the information to a 6th- to 7th-grade reading level. Last, information is then adopted for high and low information seekers.
[view this figure]
|Figure 4. Table of contents page for book on brachytherapy|
Interacting with Physicians
The physician offices are on the second and third floor of the virtual health center. The physician offices are reached through an elevator, which contains a large sign that allows the user to click on the desired office. Experts in a treatment area (ie, surgeons, radiation oncologists, a physician specializing in brachytherapy) are available to provide information about different treatment modalities. After entering the office (see Figure 5), the patient interacts with the physician by selecting different video clips from a pull-down menu. Each video clip answers specific questions about the treatment, providing anatomical and technical information. In addition, physicians discuss issues such as who is a candidate for this treatment, the likelihood of side effects, success rates, recovery time, and expected quality of life.
[view this figure]
|Figure 5. Meeting with a physician|
The Group Meeting Room
The patient may choose to participate in a group meeting of prostate cancer survivors. The members of these groups have experienced the range of treatment options. The patient may listen to the group members discuss with one another aspects of their treatment decision-making such as how they found out that they had prostate cancer, how they felt when they found they had the disease, and how they chose a treatment. Figure 6 shows members of the group discussing a wide-ranging array of topics that include sexual and incontinence problems, issues with intimacy, the effect of the disease on the partner, the influence of the spouse on treatment decision-making, experience with different treatments, and the use of alternative therapies.
[view this figure]
|Figure 6. Viewing a support-group session|
If the participant wants to take notes or jot down a comment during his exploration of the Health Center, he can call up an electronic notebook, which is available from any room. The notebook will also be helpful to jot down notes when visiting the doctor-patient module. These notes might be helpful to the patient when he consults with physicians about prostate cancer. All notes will be incorporated in the final printed report that will be generated upon exiting the program.
Exiting the Software
When participants decide to exit, the software calculates which information had been accessed and offers information that has not been viewed to patients. For example, if a patient only inquired about surgical treatment for prostate cancer, the program offers information on external beam radiation and brachytherapy for reviewing. Thus, the software will ensure that all patients have had a balanced amount of information. If the patient desires, he can complete a decision aid, designed to elicit his values and goals with regard to prostate cancer treatment  and thus assist him in making a treatment decision that is right for him. After completing the decision aid, the software will also ask the patient if he wants to return to PIES. If the answer is yes, the expert system will update the patient's results folder before the program is closed. When a participant re-enters the software he is placed in the reception area to decide which part of the health center he wishes to visit. If the participant does not intend to revisit the software, it updates the results folder, and then generates a report for the patient. This report contains a synopsis of accessed information, notes that were taken, areas the participant has not visited, and types of available information that he has not yet accessed.
Tailoring to Information Seeking Within PIES
Improved communication between physicians and patients and improved information provision about disease and its treatment are important factors in aiding patients to make treatment decisions and to cope with their disease [21,66,67]. Although, patients are generally interested in obtaining information about their disease, not all patients desire the same level of information [68,69] and thus an individualized approach to information provision is more appropriate to meet a patients information needs [70,71].
Research has identified 2 main profiles of information processing: high monitoring (information seeking) and low monitoring (information distracting). When confronted with medical information, high monitoring individuals typically process information very attentively and amplify its threatening aspects. High monitors prefer more information about their condition and its treatment, are consequently better informed, yet express greater dissatisfaction about the amount of information they receive . Overall, they demand more support and reassurance [68,72].
In contrast, low monitors demonstrate an opposite pattern; they are more likely to ignore, avoid, and minimize threatening cancer-related information that diverges from their belief that they will be fine [41,73- 76]. Consequently, low monitors prefer and are satisfied with less information about their condition, are less knowledgeable about their disease status, and tend to underestimate the risk from the disease threat.
The expert system determines if the patient is a low monitor (ie, low information seeker), one who wants only the important points of a treatment, or a high monitor (ie, high information seeker), one who wants considerable detail about a treatment therapy. The expert system directs the interactive multimedia program to provide the amount of information that the patient desires. The expert system realizes that a patient might want to be a high monitor in some areas and a low monitor in others. For example, a patient may be very interested in knowing all about the side effects of external beam radiation while wanting only the essentials about the procedure itself. To our knowledge, PIES is unique in that it allows dynamic tailoring. Consequently, as patients proceed through PIES, its expert system determines how much information patients desire about a topic and provides it.
Survey to Establish Computer Literacy Among Target Audience
Before developing PIES we conducted a survey among the target audience (ie, men diagnosed with localized prostate cancer) to establish familiarity with computers. We mailed a brief questionnaire to 675 participants of an ongoing longitudinal study of prostate-cancer treatment decision making and quality of life .
Focus Groups to Evaluate PIES
We developed a demonstration CD of PIES to showcase its features to several focus groups. The first 3 focus groups consisted of 18 prostate cancer survivors. Men were recruited from the same ongoing longitudinal study mentioned above .
Computer Literacy Among Target Audience
Of the 675 survey questionnaires mailed, 473 were returned (70% return rate). Demographic characteristics are listed in Table 1.
[view this table]
|Table 1. Demographic characteristics of PIES users|
The majority (65%) of participants reported they have a PC at home, and an additional 7% indicated they had access to a PC at a public place, such as at work or at the library. Access to a computer came with access to the Internet for 70% of the respondents. When asked to rate their experience in using a computer, 80% of patients indicated they are familiar with computers. They rated their expertise from slightly experienced (39%), moderately experienced (18%), very experienced (11%), to expert (2%). Three quarters (76%) further indicated that they had access to a family member or a friend to help them with potential computer problems. When asked about any physical problems that might prevent them from using a computer, 89% of patients reported not having any eye problems that might interfere with using a computer, and 83% indicated they have no difficulty at all in operating a pointing device such as a mouse (10% indicated having "a little problem" with a mouse). Patients who have a computer at home spend an average of 3 hours (SD = 1.5 hours) per week using it. About 20% spend more than 2 hours per day on the computer. Thus, our target audience for PIES is experienced in using a computer.
Of the 18 men, 7 (39%) were African American. We oversampled African American men into the focus groups, in part because, prostate cancer is particular prevalent among African Americans and thus the use of the program might be specifically relevant to this group. Men were on average 67 years old and had at least a high-school education (33%), a large majority was married (83%), and had completed treatment (83%). External beam radiation was chosen by 72%, 22% chose surgery, and 6% chose brachytherapy. The remaining 2 focus groups were held with spouses of prostate cancer survivors (N = 15). The women were on average 60 years old and 50% had a college or postgraduate degree. All focus group participants received $40 as a token of appreciation.
Focus Groups' Evaluation of PIES
Overall feedback on the demonstration CD was extremely positive. All ratings were performed on a 5-point scale, with higher scores indicating higher levels of interest. Patients and spouses uniformly stated that they were "very much" interested in the software (mean = 4.71; SD = 0.59; range, 3-5), and that it was "very" useful (mean = 4.71; SD = 0.47; range, 4-5). When asked whether they would prefer the software in lieu of traditional information sources, such as books or booklets, 83% stated a clear preference for the software program over print material, and 100% would prefer the program in addition to print material. On average, patients indicated that they would spend between 1 and 2 hours with the program and were willing to pay an average of $50 for it, if it were commercially available. A large majority of patients (82%) indicated that they had access to a personal computer and that they frequently use it (41% everyday, 29% 3-4 times a week, 18% 1-2 times a week); 94% had access to the Internet. In general, patients use their computer for e-mail (71%), and to obtain general (77%) and health (71%) information. Almost 60% have used the Internet to obtain prostate cancer information.
Concept of a Virtual Health Center With Rooms or Offices
Transcripts of the participants' comments revealed that participant's liked the concept of a virtual health center. Uniformly, the idea of going to rooms or offices to obtain information was found intuitive and appealing. Further, participants appreciated the variety of information that could be found in the program. The following summarizes comments by focus group members, organized by rooms.
Men and spouses found the Introduction to PIES easy to follow and indicated that it gave an excellent overview of the program. They particularly liked the possibility of accessing information in any order they liked and the program's capability of tailoring the information to their information-seeking needs. Participants also mentioned that they value an interface that mimics an interaction with a human. Some men suggested the inclusion of a guide who follows them around in the health center and who could be asked questions.
The large majority of men and spouses (90% for each) indicated that they would visit the physician's offices first, before going to any other room. Participants liked the opportunity to type in a question, which will prompt the program to retrieve the appropriate video with the physician answering the questions. However, a majority of patients also requested an overview of available physician answers. As one man stated: "After a diagnosis I didn't know what type of questions to ask. An index of available information from the physician would be very helpful." Some men also didn't like the physician sitting behind a desk, which increased the perceived distance between patient and physician. The current version of PIES places the physician in a comfortable chair in front of the desk.
Participants loved the layout of the library and found the idea of providing information in book form very appealing. They particularly appreciated the combination of written text with illustrations and short video clips. Some men voiced interest in watching video clips of surgical or seed-implantation procedures. Others, in contrast, indicated that they would not be interested in such a level of detail, providing unwitting support for our plans to tailor information to their monitoring preference. Both patients and spouses liked that some medical terms were hyperlinked to the Glossary, which provided short one-sentence explanations of the term.
In general, men showed great interest in watching video clips of prostate cancer survivors sharing their experience. In the initial format there were 3 men sitting behind a table answering questions that were keyed in by the patient. While about half of the patients appreciated the opportunity to interact with each man directly in the support group, the other half was interested in watching the men exchanging their ideas. The current version of the software includes videotapes of men discussing certain topics, such as treatment decision-making, treatment experience, and post-treatment quality of life.
Topics Specifically Mentioned in Focus Groups With Spouses
Certain topics were specifically mentioned in the focus groups with spouses. Spouses advocated for a room that provided information specific to their information needs. Topics of interest were information about nutrition (eg, soy, lycopenes), emotional support (both resources for support, as well as learning from the experience of other spouses), as well as instrumental support, particularly with caregiving after treatment. Spouses were also interested to receive information about sexual issues as it pertains to intimacy, communicating to one's spouse about sexual issues, and the use of devices to assist a patient with erectile dysfunction.
In summary, both men and spouses were enthusiastic about the program. They expressed great interest and regretted not having had such a tool when they had to make a treatment decision. When given the choice between traditional information sources such as books or pamphlets almost every participant indicated a preference for the CD-ROM. Of particular interest to patients were the physician's office and the support-group room, which offered patients expert as well as lay perspectives of prostate cancer treatments. Although men expressed proficiency with computers in general, it became clear that additional navigational and organizational help would enhance the usability of PIES.
In this paper we have introduced the development and preliminary evaluation of novel multimedia-education software for prostate cancer patients. The article started with a brief review of the major developments in computer-aided learning, particularly emphasizing research and development efforts in the area of providing (tailored and nontailored) health information and education. We then introduced the PIES architecture, described the visual interface, content development, and preliminary evaluation through focus groups.
Plans for a Virtual Information Specialist and Interactive Tutor (VISIT)
Overall, the focus group members evaluated PIES very positively and provided important suggestions about potential changes and enhancements to the program. Based on these comments and in accordance with health-behavior and communication theories [7,77] we are planning to enhance the next version of the PIES software by including a virtual information specialist and interactive tutor (VISIT). This information specialist and tutor will be designed to further enhance the interactivity between user and software. Specifically, VISIT will proactively query the user about what kind of information he desires, inform him where he can obtain the information, and will offer to retrieve it for him. VISIT will further guide the user through the program by offering relevant information that had not been accessed, thus anticipating the user's informational needs. Furthermore, while offering treatment information that could be upsetting to some patients the program will also offer emotionally-reassuring and normalizing messages.
VISIT will be represented as a health care professional (see Figure 7) through video and still photos. Once the patient enters the health center, she will introduce herself and will give an overview of the center. She will offer suggestions on how to best use the program, depending on the user's needs. For example, if a patient is interested in hearing a physician's opinion, VISIT will offer to show him the way to the doctor's office. In contrast, if the patient is interested in learning about different treatment modalities in great detail, VISIT will accompany him into the library. Once there, VISIT will query the user about which treatment options he might want to learn about first and which aspects he is most interested. The corresponding information will then be presented.
[view this figure]
|Figure 7. The virtual information specialist and interactive tutor (VISIT) at her desk|
Areas in Which VISIT Will Increase the Functionality of PIES
Provide Balanced Picture of Treatment Options
Most importantly, VISIT will ensure that the patient receives a balanced picture of all treatment options. Patients who are accessing information about only one treatment option will be queried by VISIT as to whether they are interested in receiving relevant information about other treatment modalities. VISIT will present the information or offer to take the patient to the appropriate room.
Increase Retention of Information and Provide Emotional Reassurance
VISIT is designed to increase the patient's retention of the presented information and to provide emotionally-reassuring and normalizing messages. In particular, after a module is completed VISIT will summarize the information that was given to the patient. In this context, VISIT will emphasize that prostate cancer is a common disease among men, that the tumor is slow growing and highly treatable. Furthermore, VISIT will offer information about support groups and additional information on the Internet.
Raise Patient's Perceived Self-Efficacy
Another area that will be addressed through VISIT is to raise the patient's perceived self-efficacy to deal with his condition. On the one hand, this will be achieved by providing information that is tailored to the patient's information-seeking needs. On the other hand, VISIT will provide concrete suggestions designed to enhance the patient's communication skills with his provider. VISIT will display sample interactions between patient and physician.
Short video clips depicting situations in which the patient is successful in requesting and obtaining information about his treatment options from the physician will represent these interactions. After watching these short video clips, VISIT will summarize the most important communication practices, including sample questions, that the patient can use to enhance his communication with his provider.
Suggest Completion of a Treatment Decision Aid
Finally, after the user has accessed a sufficient amount of information, VISIT will suggest the completion of a treatment decision aid. Completing a decision aid will be a 2-step process and follows the latest findings in theory and research .
In Step 1 the patient is asked to identify and rank his treatment-related values and goals. Specifically, he is asked to complete a table that will allow him to list attributes that are important for him (such as high rate of success, convenience, avoidance of pain). The patient can choose from a list of commonly-named treatment-related attributes as well as include and rank his own goals. When completed this table will consist of his own list ranked by importance of treatment-related goals.
In Step 2 the computer will carry over the ranked items into a larger table that consists of rows and columns. In column 1 the ranked items appear; the columns to the right represent the different treatment choices (eg, surgery, brachytherapy). Now the patient is asked to rate how well, in his mind, each treatment addresses his ranked attributes. Ratings will be done on a 5-point scale (1 = not at all, 5 = completely). If the patient feels that he has not enough information he can enter a question mark. This will serve as a reminder to either query PIES for further information on this issue or discuss this issue with his physician. The computer will automatically tally up the ratings and will summarize the results for him. A sample paragraph could read like this:
According to your ratings, it is very important to you to not experience urinary problems and to be able to maintain an active lifestyle. You also mentioned that you favor a high likelihood to cure the cancer, and are quite a bit worried about pain and recovery time. Your ratings suggest that you are leaning towards radiation therapy, specifically external radiation therapy. To learn more about this therapy please click here. You might also want to talk to a radiation oncologist about this treatment option.
After the user completes the decision aid, VISIT will ask whether the user would like to print any of the patient's notes before exiting the program.
Our preliminary evaluation has some limitations. First, focus group members evaluated a demonstration CD, designed to elicit input about the metaphor, its presentation, and possible content. These individuals had successfully completed treatment and thus their information needs were not as immediate as those of just-diagnosed patients. Yet, a number of survivors and spouses mentioned that they wished they had PIES while they were learning about prostate cancer. It should also be pointed out that PIES is intended to serve as an additional patient-education tool. It is not designed to substitute for personal consultations with a physician. Indeed numerous statements will be made to encourage the patient to discuss specific individual medical issues with a health care provider.
The next step in the evaluation of PIES will involve extensive usability testing with a patient sample. We will use the methodology and procedures set forth by the National Cancer Institute's own usability program (Usability.gov, ). Patients will be asked to talk aloud while exploring an alpha version of PIES, detailing what type of information they are looking for, where they think they can find it, and their reactions to it. Based on current usability standards we estimate that a small number of users (ie, approximately 5 patients) is sufficient to detect the majority of bugs. All suggestions from the usability version will be incorporated into a beta version of the program. This version will then be evaluated for efficacy in a randomized controlled trial, in which PIES will be evaluated against standard care (ie, information provided through brochures and pamphlets).
In this paper we have introduced a novel interactive multimedia system for prostate cancer education and treatment decision making. PIES is based on current psychological theory incorporating tailoring to information-seeking preferences with the latest software technology. The program is highly interactive, combining user input with animation, video, text, and still photos. Preliminary evaluation through focus groups has established the validity of our approach and has pointed to new directions to further enhance the user interface through the development of a virtual information specialist and interactive tutor.
Preparation of this manuscript was supported in part by grants from the National Cancer Institute CA6136-05, CA90904-02, and CA06927 which supports the Behavioral Core Research Facility at Fox Chase Cancer Center.
Conflicts of Interest
Screenshots of PIES (Prostate Interactive Education System): [PPT File, 3.3MB]
- . Cancer facts & figures 2003. 2003. URL: http://www.cancer.org/downloads/STT/CAFF2003PWSecured.pdf [accessed 2004 Jan 14] [WebCite Cache]
- Millikan R, Logothetis C. Update of the NCCN guidelines for treatment of prostate cancer. Oncology (Huntingt) 1997 Nov;11(11A):180-193. [Medline]
- Baker LH, Hanks G, Gershenson D, Kantoff P, Lange P, Logothetis C, et al. NCCN Prostate Cancer Practice Guidelines. The National Comprehensive Cancer Network. Oncology (Huntingt) 1996 Nov;10(11 Suppl):265-288. [Medline]
- Barry MJ, Fowler FJ, Mulley AG, Henderson JV, Wennberg JE. Patient reactions to a program designed to facilitate patient participation in treatment decisions for benign prostatic hyperplasia. Med Care 1995 Aug;33(8):771-782. [Medline]
- Diefenbach MA, Dorsey J, Uzzo RG, Hanks GE, Greenberg RE, Horwitz E, et al. Decision-making strategies for patients with localized prostate cancer. Semin Urol Oncol 2002 Feb;20(1):55-62. [Medline] [CrossRef]
- Butz BP. Expert systems. Abacus 1987;5(1):30-44.
- Miller SM, Diefenbach MA. The cognitive-social health information processing (C-SHIP) model: a theoretical framework for research in behavioral oncology. In: Krantz D, editor. Perspectives in Behavioral Medicine. Mahwah, NJ: Lawrence Erlbaum; 1998:219-244.
- Buchanan BG. In: Shortliffe EH, editor. Rule Based Expert Systems: The Mycin Experiments of the Stanford Heuristic Programming Project (The Addison-Wesley series in artificial intelligence). Reading, MA: Addison-Wesley; Jun 1, 1984.
- Anderson JR, Boyle CF, Resier BJ. Intelligent tutoring systems. Science 1985;228:456-462.
- Skinner CS, Strecher VJ, Hospers H. Physicians' recommendations for mammography: do tailored messages make a difference? Am J Public Health 1994 Jan;84(1):43-49. [Medline]
- Brown JS, Burton RR. Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science 1978;2(2):155-192. [CrossRef]
- Wenger E. Artificial Intelligence and Tutoring Systems: Computational and Cognitive Approaches to the Communication of Knowledge. Los Altos, CA: Morgan Kaufmann Pub; Oct 1, 1987.
- Polson M, Richardson JJ, editors. Foundations of Intelligent Tutoring Systems. New York, NY: Lea; May 1, 1988.
- Van Lehn K. Conceptual and metalearning during coached problem solving. In: Frasson C, Gauthier G, Lesgold A, editors. Intelligent Tutoring Systems: Third International Conference, Its '96: Proceedings (Lecture Notes in Computer Science). New York: Springer; Jun 1, 1996.
- Sears AL, Watkins SE. A multimedia manual on the World Wide Web for telecommunications equipment. IEEE Transactions on Education 1996;39(3):342-348. [CrossRef]
- Butz BP. Interactive multimedia software for a course in introductory circuits. Presented at: AACE World Conference on Educational Multimedia and Hypermedia; 1997; Calgary, Alberta, Canada.
- Edminister JA, Nahvi M. Schaum's Outline of Electric Circuits. New York: McGraw-Hill Trade; May 1, 1995.
- Mosterman PJ, Campbell JO, Brodersen AJ, Bourne JR. Design and implementation of an electronics laboratory simulator. IEEE Transactions on Education 1996;39(3):309-313. [CrossRef]
- Bergland M, Klyczek K, Lundeberg M, Mogen K, Johnson D. Case It! A project to integrate collaborative case-based learning into international undergraduate biology curricula. Presented at: 10th International Conference on College Teaching and Learning; 1999; Jacksonville, FL.
- Harger RO. Teaching in a computer classroom with a hyperlinked, interactive book. IEEE Transactions on Education 1996;39(3):327-335. [CrossRef]
- Butz BP. Freedom of choice in an intelligent tutoring system. Presented at: 1999 Meeting of the American Society for Engineering Education; 1999; Charlotte, NC.
- Butz BP. An interactive multimedia intelligent tutoring system. Presented at: 10th International Conference on College Teaching and Learning; 1999; Jacksonville, FL.
- Sleeman D, Brown JS, editors. Intelligent Tutoring Systems (Computers and People Series). New York, NY: Academic Press; 1997.
- Lewis M. Designing for human-agent interaction. AI Magazine 1998;19(2):67-78.
- Milton-Benoit J, Grosse IR, Poli C, Woolf BP. The multimedia finite element modeling and analysis tutor. ASEE Journal of Engineering Education 1998;87(5):511-518.
- Hofstetter FT. Multimedia Literacy with Student CD-ROM. New York, NY: Irwin/McGraw-Hill; Oct 3, 2000.
- Shortliffe EH, Buchanan BG. A model of inexact reasoning in medicine. Mathematical Biosciences 1975;23(3-4):351-379. [CrossRef]
- Pople HE, Myers JD, Miller RA. DIALOG: a model of diagnostic logic for internal medicine. In: Proceedings of the Fourth International Joint Conference on Artificial Intelligence. Tiblisi, Georgia: Morgan Kaufman; 1975:848-855.
- Chandrasekaran B, Gomez F, Mittal S, Smith J. An approach to medical diagnosis based on conceptual structures. In: Proceedings of the Sixth International Joint Conference on Artificial Intelligence. Tokyo, Japan. San Francisco, CA: Morgan Kaufman; 1979:134-142.
- Gorry GA, Silverman H, Pauker SG. Capturing clinical expertise. A computer program that considers clinical responses to digitalis. Am J Med 1978 Mar;64(3):452-460. [Medline] [CrossRef]
- Flynn G. Medicine in the age of the computer. Englewood Cliffs, NJ: Prentice-Hall; 1996.
- Quaglini S, Barahona P, Andreassen S. Proceedings of the 8th Conference on AI in Medicine in Europe. Cascais, Portugal: Springer; Jul 1, 2001.
- Binnie C, Spreng M. The epilepsy tutorial. Stud Health Technol Inform 2000;78:213-284. [Medline]
- Michel MS, Knoll T, Köhrmann KU, Alken P. The URO Mentor: development and evaluation of a new computer-based interactive training system for virtual life-like simulation of diagnostic and therapeutic endourological procedures. BJU Int 2002 Feb;89(3):174-177. [Medline] [CrossRef]
- Müller-wittig WK, Bisler A, Bockholt U, Los Arcos JL, Oppelt P, Stähler J, et al. LAHYSTOTRAIN development and evaluation of a complex training system for hysteroscopy. Stud Health Technol Inform 2001;81:336-340. [Medline]
- Zaharakis ID, Kameas AD, Nikiforidis GC. A multi-agent architecture for teaching dermatology. Med Inform (Lond) ;23(4):289-307. [Medline]
- Ambrosiadou BV, Ketikidis PH. DIABETOR computer aided tutoring in diabetes management. Stud Health Technol Inform 1997;43 Pt B:694-698. [Medline]
- Johnson KA, Smith JW, Dejongh M, Fischer O, Amra NK, Bayazitoglu A. RedSoar--a system for red blood cell antibody identification. Proc Annu Symp Comput Appl Med Care 1991:664-668. [Medline]
- Tambe M, Johnson WL, Jones RM, Koss F, Laird JE, Rosenbloom PS, et al. Intelligent agents for interactive simulation environments. AI Magazine 1995;16(1):15-39.
- Binik YM, Servan-schreiber D, Freiwald S, Hall KS. Intelligent computer-based assessment and psychotherapy. An expert system for sexual dysfunction. J Nerv Ment Dis 1988 Jul;176(7):387-400. [Medline]
- Agre P, Dougherty J, Pirone J. Creating a CD-ROM program for cancer-related patient education. Oncol Nurs Forum 2002 Apr;29(3):573-580. [CrossRef] [Medline]
- Guendelman S, Meade K, Benson M, Chen YQ, Samuels S. Improving asthma outcomes and self-management behaviors of inner-city children: a randomized trial of the Health Buddy interactive device and an asthma diary. Arch Pediatr Adolesc Med 2002 Feb;156(2):114-120. [Medline]
- Dragone MA, Bush PJ, Jones JK, Bearison DJ, Kamani S. Development and evaluation of an interactive CD-ROM for children with leukemia and their families. Patient Educ Couns 2002 Apr;46(4):297-307. [Medline] [CrossRef]
- Valdez A, Banerjee K, Fernandez M, Ackerson L. Impact of a multimedia breast cancer education intervention on use of mammography by low-income Latinas. J Cancer Educ 2001;16(4):221-224. [Medline]
- Neafsey PJ, Strickler Z, Shellman J, Padula AT. Delivering health information about self-medication to older adults: use of touchscreen-equipped notebook computers. J Gerontol Nurs 2001 Nov;27(11):19-27. [Medline]
- Shaw MJ, Beebe TJ, Tomshine PA, Adlis SA, Cass OW. A randomized, controlled trial of interactive, multimedia software for patient colonoscopy education. J Clin Gastroenterol 2001 Feb;32(2):142-147. [Medline] [CrossRef]
- Wardle J, Steptoe A, Smith H, Groll-knapp E, Koller M, Brodziak A. Breast self-examination: attitudes and practices among young women in Europe. Eur J Cancer Prev 1995 Feb;4(1):61-68. [Medline]
- Berridge E, Roudsari A, Taylor S, Carey S. Computer-aided learning for the education of patients and family practice professionals in the personal care of diabetes. Comput Methods Programs Biomed 2000 Jul;62(3):191-204. [Medline] [CrossRef]
- Hornung RL, Lennon PA, Garrett JM, Devellis RF, Weinberg PD, Strecher VJ. Interactive computer technology for skin cancer prevention targeting children. Am J Prev Med 2000 Jan;18(1):69-76. [Medline] [CrossRef]
- Block G, Miller M, Harnack L, Kayman S, Mandel S, Cristofar S. An interactive CD-ROM for nutrition screening and counseling. Am J Public Health 2000 May;90(5):781-785. [Medline]
- Consoli SM, Ben Said M, Jean J, Menard J, Plouin PF, Chatellier G. Benefits of a computer-assisted education program for hypertensive patients compared with standard education tools. Patient Educ Couns 1995 Sep;26(1-3):343-347. [Medline] [CrossRef]
- Gustafson DH, Bosworth K, Hawkins RP, Boberg EW, Bricker E. CHESS: a computer-based system for providing information, referrals, decision support and social support to people facing medical and other health-related crises. Proc Annu Symp Comput Appl Med Care 1992:161-165. [Medline]
- Gustafson DH, Hawkins R, Boberg E, Pingree S, Serlin RE, Graziano F, et al. Impact of a patient-centered, computer-based health information/support system. Am J Prev Med 1999 Jan;16(1):1-9. [Medline] [CrossRef]
- Mctavish FM, Gustafson DH, Owens BH, Wise M, Taylor JO, Apantaku FM, et al. CHESS: An interactive computer system for women with breast cancer piloted with an under-served population. Proc Annu Symp Comput Appl Med Care 1994:599-603. [Medline]
- Kreuter MW, Strecher VJ, Glassman B. One size does not fit all: the case for tailoring print materials. Ann Behav Med 1999;21(4):276-283. [Medline]
- Cook TD, Flay BR. The temporal persistence of experimentally induced attitude change: an evaluative review. In: Berkowitz L, editor. Advances in Experimental Social Psychology. New York: Academic Press; 1978, Vol. 11.
- Butow PN, Dunn SM, Tattersall MH, Jones QJ. Patient participation in the cancer consultation: evaluation of a question prompt sheet. Ann Oncol 1994 Mar;5(3):199-204. [Medline]
- Petty RE. The importance of cognitive responses in persuasion. Advances in Consumer Research 1977;4:357-362.
- Skinner CS, Campbell MK, Rimer BK, Curry S, Prochaska JO. How effective is tailored print communication? Ann Behav Med 1999;21(4):290-298. [Medline]
- . Authorware [computer program]. URL: http://www.macromedia.com/ [accessed 2004 Jan 14] [WebCite Cache]
- . About PDQ. URL: http://www.cancer.gov/cancerinfo/pdq/treatment/prostate/patient/#Section_AboutPDQ_14 [accessed 2004 Jan 14] [WebCite Cache]
- . Home page. URL: http://www.cancer.gov/ [accessed 2004 Jan 14] [WebCite Cache]
- . Cancer Information Service. URL: http://cis.nci.nih.gov/ [accessed 2004 Jan 14] [WebCite Cache]
- . Medline. URL: http://www.ncbi.nlm.nih.gov/PubMed/ [accessed 2004 Jan 14] [WebCite Cache]
- O'connor AM, Rostom A, Fiset V, Tetroe J, Entwistle V, Llewellyn-thomas H, et al. Decision aids for patients facing health treatment or screening decisions: systematic review. BMJ 1999 Sep 18;319(7212):731-734 [FREE Full text] [PMC] [Medline]
- Brandt B. Informational needs and selected variables in patients receiving brachytherapy. Oncol Nurs Forum 1991 Sep-Oct;18(7):1221-1227 discussion 1227-1229. [Medline]
- Ford S, Fallowfield L, Hall A, Lewis S. The influence of audiotapes on patient participation in the cancer consultation. Eur J Cancer 1995 Dec;31A(13-14):2264-2269. [Medline] [CrossRef]
- Miller SM. Monitoring versus blunting styles of coping with cancer influence the information patients want and need about their disease. Implications for cancer screening and management. Cancer 1995 Jul 15;76(2):167-177. [Medline]
- Leydon GM, Boulton M, Moynihan C, Jones A, Mossman J, Boudioni M, et al. Faith, hope, and charity: an in-depth interview study of cancer patients' information needs and information-seeking behavior. West J Med 2000 Jul;173(1):26-31 [FREE Full text] [Medline] [CrossRef]
- Degner LF, Sloan JA. Decision making during serious illness: what role do patients really want to play? J Clin Epidemiol 1992 Sep;45(9):941-950. [Medline] [CrossRef]
- Forrow L, Wartman SA, Brock DW. Science, ethics, and the making of clinical decisions. Implications for risk factor intervention. JAMA 1988 Jun 3;259(21):3161-3167. [Medline] [CrossRef]
- Ong LM, Visser MR, Van Zuuren FJ, Rietbroek RC, Lammes FB, De Haes JC. Cancer patients' coping styles and doctor-patient communication. Psychooncology 1999;8(2):155-166. [Medline]
- Miller SM, Mangan CE. Interacting effects of information and coping style in adapting to gynecologic stress: should the doctor tell all? J Pers Soc Psychol 1983 Jul;45(1):223-236. [Medline] [CrossRef]
- Miller SM, Roussi P, Altman D, Helm W, Steinberg A. Effects of coping style on psychological reactions of low-income, minority women to colposcopy. J Reprod Med 1994 Sep;39(9):711-718 [FREE Full text] [Medline]
- Miller SM, Shoda Y, Hurley K. Applying cognitive-social theory to health-protective behavior: breast self-examination in cancer screening. Psychol Bull 1996 Jan;119(1):70-94. [Medline] [CrossRef]
- Tercyak KP, Lerman C, Peshkin BN, Hughes C, Main D, Isaacs C, et al. Effects of coping style and BRCA1 and BRCA2 test results on anxiety among women participating in genetic counseling and testing for breast and ovarian cancer risk. Health Psychol 2001 May;20(3):217-222. [Medline] [CrossRef]
- Rostom A, O'connor A, Tugwell P, Wells G. A randomized trial of a computerized versus an audio-booklet decision aid for women considering post-menopausal hormone replacement therapy. Patient Educ Couns 2002 Jan;46(1):67-74. [Medline] [CrossRef]
- . Usability.gov. URL: http://www.usability.gov/ [accessed 2004 Jan 14] [WebCite Cache]
|PIES: prostate interactive education system|
|VISIT: virtual information specialist and interactive tutor|
Edited by G. Eysenbach;
submitted 27.10.03; peer-reviewed by G Kreps, EE Anderson, J Reis; comments to author 06.11.03; revised version received 22.12.03; accepted 01.01.04; published 21.01.04|
Please cite as:
Diefenbach MA, Butz BP
A Multimedia Interactive Education System for Prostate Cancer Patients: Development and Preliminary Evaluation
J Med Internet Res 2004;6(1):e3
BibTeX, compatible with BibDesk, LaTeX
RIS, compatible with RefMan, Procite, Endnote, RefWorks
Refer, compatible with Endnote
Add this article to your Mendeley library
Add this article to your CiteULike library
Add this article to your Connotea library