The World Wide Web has led to the rapid growth of medical information and continuing medical educational offerings. Ease of access and availability at any time are advantages of the World Wide Web. Existing physician-education sites have often been designed and developed without systematic application of evidence and cognitive-educational theories; little rigorous evaluation has been conducted to determine which design factors are most effective in facilitating improvements in physician performance and patient-health outcomes that might occur as a result of physician participation in Web-based education. Theory and evidence-based Web design principles include the use of: needs assessment, multimodal strategies, interactivity, clinical cases, tailoring, credible evidence-based content, audit and feedback, and patient-education materials. Ease of use and design to support the lowest common technology denominator are also important.
Using these principles, design and develop a Web site including multimodal strategies for improving chlamydial-screening rates among primary care physicians.
We used office-practice data in needs assessment and as an audit/feedback tool. In the intervention introduced in 4 phases over 11 months, we provided a series of interactive, tailored, case vignettes with feedback on peer answers. We included a quality-improvement toolbox including clinical practice guidelines and printable patient education materials.
In the formative evaluation of the first 2 chlamydia modules, data regarding the recruitment, enrollment, participation, and reminders have been examined. Preliminary evaluation data from a randomized, controlled trial has tested the effectiveness of this intervention in improving chlamydia screening rates with a significant increase in intervention physicians' chlamydia knowledge, attitude, and skills compared to those of a control group.
The application of theory in the development and evaluation of a Web-based continuing medical education intervention offers valuable insight into World Wide Web technology's influence on physician performance and the quality of medical care.
The World Wide Web (Web) provides a delivery system for conveying complex, structured information to a large number of users without the barriers of time and geography [
In 2000, 96 continuing medical education sites were available; in 2001 this number had more than doubled to 209, with 18263 hours of continuing medical education credit offered online [
Because of the potentials for high costs and adverse consequences of poor performance, medical education represents a major category of lifelong education [
Situated learning, andragogy, and cognitive flexibility theories are examples of cognitive learning theories relevant to the design of physician education. Situated learning focuses on the social nature of cognition and the importance of authentic situations to learning; specifically this requires settings and applications to be relevant to the daily life of the learner [
Several theorists have applied constructs specifically to the problem of how physicians in clinical practice learn and change.
First, motivation for physician learning has been linked to the nature of the problem, most frequently a specific patient problem as physicians seek information to deal with uncertainty in the clinical encounter: surprise, stress, and cognitive dissonance in this context may lead to information seeking [
Second, problem-based learning (PBL), a cognitive learning strategy, has been important in organizing undergraduate and graduate medical-education curricula but has not yet become the standard format for CME [
Third, application of a stages-of-change model to the question of how physicians learn and change has been suggested [
Fourth, Schoen's model has described a process by which physicians reflect on their daily practice in order to continue to learn over time; practitioners engage in situated action until their expectations are not met and they experience a breakdown in their current work situation [
Finally, Fox and others have proposed that learning and change resulting from information-seeking behavior varies from stage to stage in a multistage process [
Not only is it important to examine which learning theories are the most applicable to medical education, but also which media or educational delivery systems are most supportive of relevant theories. Since situated learning focuses on the social nature of cognition and the importance of authentic situations to learning, computer-assisted and Web-based instruction are ideal formats for CME since they can tailor the learning process to the individual student by providing teaching and support in response to the individual's immediate needs. These forms of instruction can also bring experts to the learner, demonstrating clinical reasoning skills.
Learning through the Web can also be enhanced by immediate repetition, which helps the learner make new knowledge and skills explicit. These aspects put the student at the center of the learning processes, as suggested by situated learning theory, and make many resources available through a large number of different learning pathways and possibilities [
While not specifically related to medical education, a recent investigation of research involving the development of Web-based instruction was conducted to determine which instructional design models or approaches had been adopted for the design of Web-based instruction [
In considering the appropriate theoretical context for the design of an intervention to improve physician performance in screening for chlamydia, we drew on the work of cognitive theorists, specifically work on the processes of assimilation and accommodation, and on stages of learner's development and readiness for learning. Because of its direct application to problem solving, we also used cognitive theoretical frameworks that suggest the learner control the sequence and pace of instruction, and that the instruction be individualized to participant needs.
As early as the 1970s, evidence suggested that traditional CME programming was not effective in facilitating changes in physician performance and changes in patient-health outcomes [
Most recently, Davis and colleagues summarized the results of 14 randomized, controlled trials of physician education conducted between 1993 and January 1999, concluding, from this review at least, that didactic teaching sessions do not appear to be effective in changing physician performance [
Across various reviews examining the effectiveness of interventions aimed at influencing physician behavior, the use of multiple interventions has been more successful than the use of a single episodic intervention [
Others have used feedback to providers as an intervention to improve physician performance-with the assumption that knowledge of one's own performance, together with the ability to compare this performance against some reference level (internal or external) will facilitate improvement [
Based on a MEDLINE literature search from June 2000 through June 2002, there have not yet been enough rigorous trials to determine the effectiveness of Web-based courses in improving physician performance and patient-health outcomes, therefore, meta-analyses have not been conducted. However, from the evidence of effectiveness in other reviews [
We abstracted the following Web-design principles for physician-education Web sites from learning and change theories, as well as from the evidence of what works in continuing medical education:
Office-practice data as needs assessment
Multimodal strategies
A series of modules rather than one single, episodic educational event
Contextual learning in the form of clinical cases
Tailoring based on individual responses
Interactivity
Audit and feedback
Evidence-based content
Credibility of the organization providing the Web site and instruction and of any agency providing grant support for the education or site
Patient-education materials
Ease of use of the site and ease of navigation
Design for the "lowest technological denominator" in hardware.
Although the Cochrane Collaborative has produced overall reviews of the effectiveness of CME interventions, audit and feedback, live conferences, and academic detailing, there has been too little overall evaluation using randomized, controlled trials of online-course interventions to produce a review of their effectiveness. Individual studies have tended to be descriptive and have focused on participant reactions rather than improvements in physician performance and changes in patient-health outcomes. For example, a descriptive study of online interactive pathology case studies details the interactive format of CME cases that allow participants to submit immediate comments or criticism to case authors and to receive immediate feedback on their own performance; these features are normally unavailable in traditional CME courses. The evaluators note that the dynamic environment of the Web allows development of flexible forms of CME for the physician [
A descriptive evaluation of a hybrid delivery system was conducted [
A Web-based tutorial system was compared to a print tutorial system with residents for the management of care following acute myocardial infarction. Immediate post-test scores were similar in both groups, but Web users spent less time studying, producing greater learning efficiency (
Examining the same Web-based system used with residents for the management of care following acute myocardial infarction, on average, users accessed less than half of the guideline passages and little of the graphic evidence. Greater use of guideline passages was correlated with greater immediate learning, but use of graphic evidence was not. Authors commented that further research is needed to integrate clinical-trial evidence with guideline-based education [
A pilot evaluation of a Web-based curriculum reviewed occupational and environmental health or medicine components of 2 undergraduate degree courses and 2 postgraduate courses including interactive components; 12 students achieved the main learning objectives. Participants valued the flexibility, timeliness, efficiency, and breadth of access to relevant information offered by the Web [
Most Web-based materials have not been subjected to external assessment for quality. An online questionnaire was developed covering general suitability, local suitability, user interface, educational style, and a general review-and was piloted in 3 subject areas: general chemistry, radiology, and medical physics, focusing on undergraduate teaching. The evaluation methodology was found to work well for highly-structured and formal content and may have value in helping those developing undergraduate curricula to identify appropriate Web-based materials for integration into the curricula [
While these studies are useful as formative evaluations of Web courses, rigorous summative evaluations are needed. Ward notes that the Web provides powerful tools for learning medical education and will alter how the discipline is taught; for the drive to incorporate such technologies threatens to outstrip an overall understanding of how they can be used most effectively [
In collaboration with a large national health maintenance organization, a needs assessment of current practice patterns related to screening for chlamydia was conducted on a national sample of physicians from 16 states. A series of three 1-hour Web modules was designed to increase primary-care physician screening rates in the population of women ages 16 to 26 years. A physician was identified as eligible for participation if the physician's office participated in a specific health plan, if the physician's office had at least 10 sexually-active patients between the ages of 16 and 26 years, and if the physician had Internet and e-mail access. Physicians were recruited to participate by fax and FedEx communications. Physicians were notified of the online availability of each module by e-mail announcements as well as a series of e-mail reminders, all containing the link to the modules within the e-mail messages. The program invitation was received by 3067 physicians. Upon complete login, physicians were randomly assigned to either the intervention arm or the control arm of the study.
Baseline data demonstrated an average chlamydial screening rate of less than 20%, with the lowest rates in the group of sexually-active young women ages 21 to 26 years and higher rates in the 16 to 20 year-old group. Screening rates and the process used to abstract them from administrative data have previously been reviewed (Ray M, et al. Unpublished data, 2001).
Based on the evidence that single CME events are unlikely to improve physician performance and enhance patient-health outcomes, and evidence of the "decay effect" (a reduction of effect over time) of interventions, we used a series of 3 modules rather than 1. To enhance our theoretical chances of effect, each module would use a multimodal approach. Each module included the following components:
Today's cases: vignette series of primary care cases of young women 16-26 years of age
My screening data: audit and feedback at the office level of chlamydia screening data
Help for my office: patient brochures and physician screening guidelines.
Chlamydia module I course home page
The section titled "Today's cases" was developed based: (1) on studies demonstrating that physician information seeking on the Web is most-frequently connected to searching for information related to a specific patient problem and (2) on adult learning principles that focus on the relevance of instruction to an adult's work and life. Theoretical constructs from problem-based learning and from situated learning theory are central to this design in creating an authentic contextual process through the use of cases and in putting the learner at the center of the learning processes. The Schoen cyclical model of action, breakdown, and reflection was selected as the central focus of the chlamydia intervention. This parallels a physician's motivation to seek information for purposes of patient problem-solving [
More prominently featured in earlier forms of behavioral theory-based computer-assisted instruction, tailoring and branching of responses to meet individual needs has not been broadly applied to the development of Web-based instruction. Modules were developed with individualized responses to multiple case-based and practice-based questions, leading to over 300 possible permutations of pathways throughout individualized modules. Using methods reported by Kreuter et al [
In addition to physician case responses as a means of tailoring and remediation, we used the theoretical constructs related to stages of change to develop tailored pathways throughout the modules. A series of attributes were attributed to various stages of change. For example, if a learner considered the overall prevalence of chlamydia to be less than 1% or if the learner did not take sexual histories, he or she would be categorized as precontemplative and branched to pathways designed to heighten awareness of the growing prevalence of chlamydia or the importance of taking a sexual history. If, however, the learner demonstrated skill in taking a sexual history and reported updating sexual histories at each visit with young female patients, the learner would be branched to pathways designed to reinforce current skills and evidence that supports the need to continue to focus on these activities [
Dick and Carey have noted the frequent lack of emphasis in instructional design on practice and rehearsal of skills, resulting in a lack of transference of skills into practice [
The collaboration of a university medical center with a large national health maintenance organization has significantly enhanced the development of this Web-based intervention in several ways. First, administrative data provided a solid basis for needs assessment, as well as baseline measurement of physician performance in chlamydia screening, by identifying sexually-active women between the ages of 16 and 26 years. Second, a list of physicians was linked to patients as the primary care providers for these patients for the purposes of intervention. Administrative data can then be used to provide data, following the intervention, on its effectiveness. For the purposes of the intervention itself, data at the individual office level, as well as the regional level, could be provided to participants in the Web intervention. Using a Structured Query Language (SQL) database server and participant log-on criteria, each participant was able to access office-level data as a means of audit and feedback. A depiction of "Screening performance for my office" is in
Chlamydia module I "screening performance for my office" page
Initial data demonstrated that this page was the most-frequently visited page. In the evaluation survey, this module element was cited as the most useful.
Core messages were developed based on a new set of guidelines developed by the US Preventive Task Force [
Sexually-active women between the ages of 16 and 26 years are at highest risk for chlamydial infections that can lead to pelvic inflammatory disease, increased risk of HIV infection, and infertility.
New urine-based testing allows screening for chlamydial infection without a pelvic examination.
Chlamydial infections can be effectively treated with a 1-dose antibiotic, increasing the likelihood of adherence to treatment [
Case answers were supported by evidence from the literature and were referenced to provide physicians with additional sources of information.
In the section labeled "Help for my office" the following materials have been included to support the physician in translating new knowledge and skills into the practice environment:
Patient education brochures on sexually transmitted diseases (STDs) and chlamydia, including Spanish and English versions
Brief summary of physician guidelines
Access to full text of new screening guidelines
Confidentiality policy regarding disclosure of teen information to parents.
In the formative evaluation of the first 2 chlamydia modules, data regarding the recruitment, enrollment, participation, and reminders have been examined. From the initial recruitment period (February 1, 2002, through October 2, 2002) there were 3067 primary care physicians in 1045 offices that received fax materials recruiting their participation. Of those, 463 physicians registered by fax for the course and provided their e-mail addresses. Although 210 physicians logged on, only 180 physicians completed the first module. Among the physicians completing the module, 92 were randomly assigned at the time of log on to the intervention group and 89 to the control group. The second module was completed by 134 physicians. For the third and forth modules, 96 and 61 physicians participated, respectively.
Physicians were asked to assess the usefulness of particular course elements: cases, screening information, "My screening practices," and "Help for my office."
Intervention physician perceptions of "usefulness" of course elements
|
|
|
|
|
Cases | 41 | 31 | 31 | NA |
Screening information | 41 | 22 | 7 | NA |
My screening practices | 5 | 7 | 3 | NA |
Help for my office | 2 | 3 | 0 | NA |
* NA = Not Available
Chlamydia knowledge, attitudes, and skills of intervention physicians and control physicians were measured and compared following the intervention. A 21-item post-test was conducted following participation in the fourth module. The mean number of correct responses was calculated for each group. The mean number of correct answers to the content questions collected from module IV demonstrates a significant difference between physicians in the intervention and control groups. The control-group mean score of correct answers was 78.3 compared to 93.0 for the intervention group. A paired t test calculation determined the difference to be statistically significant at an alpha level of .0003.
We are conducting a further evaluation of the chlamydia screening Web-based intervention design. Additional outcomes are the differential improvement in screening rates of the 2 study arms as ascertained from administrative data. Patient-level multivariable analyses will adjust for the extrabinomial variation resulting from patients being nested within physician offices from the group randomized design. Screening rates from the calendar year of 2001 will be compared with those from 2002 to determine both within-group and between-group differences.
Analyses are scheduled to be complete by the end of 2003. Two other evaluation studies of this intervention have been initiated; the intervention is being adapted to a project designed to improve the diagnosis and management of glucocorticoid-induced osteoporosis and a second project with the goal of increasing the adoption of multiple secondary prevention guidelines for the post myocardial infarction patient.
While traditional instructional design models have been applied to the creation and development of Web activities, the failure to apply cognitive learning theory to Web design creates a challenge for those developing Web-based instruction for physicians. The need for rigorous evaluation of Web-based interventions has been well documented but has yet to produce enough evidence to create a performance benchmark. Application of theory and rigorous evaluation will be crucial over the next decade to those who wish to use the Web to influence physician performance and the quality of medical care.
This project was supported by grant number U18 HS 11124 from the Agency for Healthcare Research and Quality.
None declared.