Developing Countries Must Invest in Access to Information for Health Improvements

Background: Use of information technology in information acquisition, especially MEDLINE on CD-ROM and online, has been evaluated in several localities and regions, especially in the advanced countries. Use of MEDLINE on CD-ROM is still very poor among the medical students of the University of Lagos, Lagos, Nigeria, due to lack of awareness, insufficient personal computers, nonperiodic training, and the high cost of using the facility. Due to financial constraints, MEDLINE online and sufficiently-networked computer systems are not available. Objective: To report on the situation in Nigeria, a developing country, so as to compare the current awareness of searching MEDLINE on CD-ROM among the medical students at the University of Lagos with the awareness of their overseas' counterparts. This is the first step toward setting up an online PubMed search as well as expanding the computer systems and network. Methods: Essentially based on cross-sectional proportional sampling using structured questionnaires, in-depth interviews, and focus-group discussions among the medical students and library staff. The study involved the medical students in their second year to sixth (final) year of study. Results: Of the 250 students interviewed, 130 (52%) were aware of MEDLINE on CD-ROM searches as a means of information retrieval. Only 60 (24%) had used MEDLINE on CD-ROM — 2% had used MEDLINE on CD-ROM more than 9 times; 4%, 7 to 9 times; 8%, 4 to 6 times; and 10%, 1 to 3 times. Of the students who used MEDLINE on CD-ROM search, 22% used it in preparing for examinations, 24% in research, 6% in patient care, and 26% in preparation of assignments and clinical cases. Lack of awareness (52%) and cost of undertaking MEDLINE on CD-ROM search (46%) were identified as important factors that discouraged the use of MEDLINE on CD-ROM. Conclusions: Though the above factors were recognized as important, it was concluded that the reasons for the poor use of MEDLINE on CD-ROM are multifactorial. Poor use of MEDLINE on CD-ROM could be attributed to these critical underlying factors: nonavailability of networked personal computers, which should be connected to a central server; lack of mandatory assignments to the medical students that would specifically require use of MEDLINE on CD-ROM; financial constraints on the university management; and infrequent periodic orientation on use of MEDLINE on CD-ROM. It was therefore suggested that the number of personal computers should be increased and that the library staff should periodically train the preclinical and clinical medical students in searching MEDLINE on CD-ROM. These steps would enable the medical students to benefit from online PubMed searching when it becomes fully operational in the future. (J Med Internet Res 2003;5(1):e7)   doi:10.2196/jmir.5.1.e7

That many developing countries, especially in sub-Saharan Africa, have low health status is not news. What is of interest and germane to the process of public health development and reform in these countries is sustainable ideas and proposals for health improvements. Among the many proposals discussed in international health circles, access to appropriate and credible information is topical. Why is this so?
Systematically-harnessed information and communication technologies have been shown to improve the health of populations in many developed countries through empowerment of those who access and use information -from the simple homemaker and working mother to the highest-ranking policymaker. These technologies have also been shown to democratize the public space by fostering freedom of choice and expression as well as rapid access to and sharing of information. These highly-cherished values of participatory democracy have in turn clearly helped to engender better health in individuals and communities. Fascinating and beneficial as these technologies may sound, the deep-rooted factors militating against access to information and the daunting challenges thereto in developing countries are well known [1,2]. Developing countries have generally not invested in access to information as much as they should, even within their available means. Indeed, they are being left behind (the so-called digital gap or digital divide). Very little benefit is accruing to developing countries from the incredible and bountiful digital world. But these countries need to benefit for many good reasons. For example, Ade Lucas, Adjunct Professor of Public Health at Harvard School of Public Health, recently addressed the daunting health challenges existing in most developing countries today. He demonstrated clearly how available evidence-based information -when accessed and used appropriately in the public-policy domain -can make a difference to policy choices and decision-making in the development and implementation of HIV/AIDS prevention programs in developing countries [3]. Regrettably, for many reasons that are often recounted but not always tenable, most developing-country governments and people have not harnessed available information and communication technologies to improve public health in their respective countries. In volunteering a solution, Lucas asked a rhetorical question: "How can African governments tackle their daunting health problems especially in the face of limited financial resources?" His answer: "they must adopt health policies that are based on good information and must use the most [evidence-based] and cost-effective interventions in their programmes" [3].
There is no question that good and evidence-based information is available worldwide and that the Internet is providing both the tools and platform for low-cost, area-wide, and effective dissemination and retrieval of such information. Availability of information is one thing, access to and use of the available information is another. Behind Lucas' advocacy is a clear call for a rethink of the public-health policy-making process in most developing countries where it is not uncommon to misplace priorities and place emphasis on tertiary care and high technology procurement to the detriment of primary health care -including appropriate information provision.
For developing countries to achieve the benefits of access to health information, they must invest strategically in information production, gathering, storage, dissemination, and public health literacy promotion. In most developing countries, where the total per capita spending on health is less than $15 per year [4], is this an economically viable investment? The answer must be a resounding "Yes" bearing in mind the multiplicative empowering impact of appropriate information on millions of people, compared to the few hundreds or thousands that white-elephant tertiary hospitals may benefit per time frame.
How may the rethink for investment in access to health information be achieved in developing countries? Herein lies a challenging role for development and multilateral institutions and agencies like the World Health Organization (WHO) and other United Nations (UN) organizations, as well as the various research and leadership systems in developing countries. Given the global impact of UN multilateral institutions in development matters, a call for them to enunciate policies and foster programs

Introduction
The application of information technology has revolutionized the mode of acquisition, storage, and dissemination of information. Today, computer access to the medical literature has become more convenient than ever before with MEDLINE on CD-ROM and online. Computerized literature searching enables the user to efficiently identify germane articles and research studies. Physicians can also learn to use a computerized literature-searching system and efficiently retrieve materials related to patient care [1,2]. In medical education, the literature also functions as source material for conference presentations, course and licensing examinations, teaching assignments, and research studies [3][4][5]. While educators agree that medical students should learn to use MEDLINE for clinical application, there is a lack of consensus on an optimal level of exposure to this resource during training that will result in sustained usage [6].
Although the Medical Library of the College of Medicine of the University of Lagos, Lagos, Nigeria, subscribes to a considerable number of worldwide medical journals, the unavailability of these journals when they are supposed to be available impedes the supply of information [7]. The problem of supply of current journals is due largely to cost, and this is where MEDLINE on CD-ROM becomes useful in the provision of primary information pending the arrival of the journals.
Online MEDLINE facilitates the retrieval of information from over 4500 journals published throughout the world. The availability of the full text of publications and articles implies that this process, with its recognizable advantages, will be useful in the training of medical students. Over the years, the Medical Library of the University of Lagos has made efforts toward providing MEDLINE both on CD-ROM and online in addition to other facilities that would enhance information acquisition. However, lack of funds to purchase the required computers and to network the computers with online connections limited these efforts to the provision of MEDLINE on CD-ROMs that do not require online connection. Since the personal computers are few, networking was also not available. The extent to which the medical students utilize the available facilities for MEDLINE on CD-ROM in their training is not known. We therefore investigated the awareness and use of MEDLINE on CD-ROM by the medical students of the University of Lagos, College of Medicine in comparison with what exists overseas. This is the first step toward the introduction of online PubMed search when funds become available.

Study Area
The Medical Library of the College of Medicine, University of Lagos, was used for this study. This library serves the entire medical community made up of undergraduate (medical, dental, pharmacy, and basic medical sciences) and postgraduate students (MSc, MPhil, and PhD levels), resident doctors, nurses, medical laboratory technologists, health workers, and the academics. Generally, the medical library serves the Lagos University Teaching Hospital (LUTH), the School of Nursing and Midwifery, Post-Basic Nursing, the School of Medical Laboratory Technology, and the School of Medical Records and Statistics.
The College of Medicine of the University of Lagos is made up of: the School of Basic Medical Sciences, the School of Clinical Sciences, the School of Dental Sciences, the School of Pharmaceutical Sciences, and the Institute of Child Health and Primary Health Care. Only the medical students enrolled for the Bachelor of Medicine and Surgery (MBBS) or the MD program were used in this study.
The Medical Library usually conducts a one-week orientation program for newly-enrolled medical students. During this period, the students are trained in effective ways of using the library, including using MEDLINE on CD-ROM.

Study Group
The study was conducted between June and September 2002 among the medical students of the College of Medicine, University of Lagos. The randomly-selected medical students were in the second year to sixth (final) year. The premed class was not studied, because the students were located in another campus and were yet to be enrolled in medical school.
The sampling size of this study was 230 students. A proportional sampling unit was selected from each class to ensure unbiased representation. For this study, 340 questionnaires were administered. Structured questionnaires were administered to each student. This was followed by focus-group discussions and in-depth interviews with key informants among the studied population. The focus group and key informants were randomly selected from the student population. Information was also obtained from designated class representatives of the students and library staff. Clarification that the study was not intended to punish non-users of MEDLINE on CD-ROM -the names of the students were not required in the questionnaireenhanced participation. This step removed bias from the responses provided by the medical students.

Results
Of the 340 questionnaires that were administered, 250 were duly completed. This represented a response rate of 73.5%. The sample size of 230, which was used in the study, could be taken as 100% response in principle. The study selected: 40 students in the sixth year class (16%); 65 in the fifth year class (26%), 25 in the fourth year class (10%); 90 in the third year class (36%) and 30 in the second year class (12%).
Of the 60 students who had used MEDLINE on CD-ROM at one time or the other, their purposes were: preparing for examination, 22%; personal interest, 20%; research, 24%; patient care, 6%; and preparation of assignments or clinical cases, 26% (percentages are based on the 60 students who had used MEDLINE on CD-ROM).
Lack of awareness and the cost of MEDLINE on CD-ROM searching were the major factors responsible for its poor use ( Table 1). The majority of the respondents who had used MEDLINE on CD-ROM printed copies of abstracts. The students who had used MEDLINE on CD-ROM would prefer to use it again because it saves time while giving them the opportunity to conduct as many searches as possible within a short time.

Focus Group Results
Focus Group discussions and interview of key informants exposed 3 issues:

Examples of responses:
• "We were always told to look up journals in the journal section for new information" • "We have never been told to use MEDLINE; any way, there is so much to read in our text-books" • "We are used to hard copies of texts and journals" • "Though we received some orientation on how best to use the library during our first year of enrolment in the medical school, the trend has always been the use of text-books and journals and attend clinics regularly to read case notes" • "MEDLINE is needed by the postgraduate students and resident doctors because they are doing serious research"

Cost of undertaking MEDLINE search on CD-ROM
They agreed that "the cost and waiting time, which could take several days if a particular class would have to seek information in a given time, made us to always use the available journals and then pay for photocopies." The waiting time is due to the limited number of personal computers, which are not networked.

Insufficient Personal Computers and Lack of Periodic Orientation
Examples of responses: • "Access is somewhat restricted because of the limited number of PCs; thus, we the undergraduate will have to compete for the available computers with the Postgraduate students".
• "We will be happy to have periodic orientation on MEDLINE use as well as having more computers to make the training qualitative"

Discussion
The advent of information technology, including the use of MEDLINE facilities, has, over the years, facilitated the acquisition of information. This is diametrically different from manual searches using Index Medicus and other bibliographies. To a large extent, MEDLINE (CD-ROM and online) is fully integrated into the university system in developed countries but is yet to be fully utilized in developing countries. The reasons for this poor utilization range from structural inadequacies to lack of awareness.
The usefulness of MEDLINE on CD-ROM and online are obvious and have been variously reported, though they differ from one institution to another [3][4][5]8,9]. Information acquisition is therefore difficult in developing countries where the referred journals are not available, thus making it imperative for universities to acquire new technologies such as having a highly-networked computer system that has online connections [7].
The results of this study showed that use of MEDLINE on CD-ROM is poor among the medical students. The reasons for this poor use are multifactorial. First is the attitude of the students toward MEDLINE CD-ROM use -which has made reorientation difficult. In-depth interviews revealed that the students, in most cases, are contented with printed materials available on the shelf. The students' use of printed materials is strengthened by the use of texts by students in the preclinical class (first and second year) in preparing for examinations (including in-course assessments). Our findings on use of MEDLINE on CD-ROM therefore differed from the 87% use of the computerized literature-searching system for patient write-ups by medical students of the Wright State University School of Medicine (WSUSOM), Ohio, USA [2]. Unlike the system in WSUSOM that enabled the user to have the full text of a journal article or textbook in addition to citations and abstract [2], our system only provides abstracts. The students interviewed considered abstracts inadequate.
However, the factors responsible for poor use of MEDLINE CD-ROM (Table 1) are important and crucial. Most important of these factors were lack of awareness and the cost of undertaking searches of MEDLINE on CD-ROM. This finding is consistent with the recognized factors found to discourage database use: cost, availability, and awareness [10]. Though these factors were theoretically not present in the study situation at the Pritzker School of Medicine of the University of Chicago, Illinois, USA, Kaluzsa [10] reported that all 3 factors still discouraged MEDLINE use for a significant portion of the student body. The results from our study were different when compared to what was obtained among students enrolled in the undergraduate program of the McMaster University Faculty of Health Sciences, Canada [11]. In this study -which was on the knowledge and use of information technology (competence, access, usage, and perceptions of the need for training) -there was a progressive rise in reported information technology access and use for 3 years. Self-service MEDLINE use on CD-ROM was 65%, 75%, and 89% for the 1987, 1988, and 1989 classes respectively [11].
Our study was also different from the results obtained by Osman and Muir [12] among 144 third-year students at the University of Edinburgh, Scotland, UK They reported that 22% of the students had never used the university-library computerized catalog and 43% had never carried out a MEDLINE search using the library's CD-ROM. This study in the UK [12] also asserted that medical students who have not acquired basic computer information-technology skills by the third year of undergraduate training are unlikely to do so in the final hospital-based years. In our study, 90 (36%) of the sampled group were from the third year class and only 12 (13.3%) of those 90 had used MEDLINE on CD-ROM.
It costs about US $0.11 to print a copy of an abstract. However, viewing an abstract does not require any form of payment. Even so, viewing would be difficult because the systems are not sufficient to serve those who would wish to pay for a literature search. The nonavailability of networked personal computers was identified during in-depth discussion with key informants as a critical structural inadequacy that is responsible for the nonuse of MEDLINE in the university. Furthermore, lack of awareness appears to be an important factor in the nonuse of MEDLINE.
In most cases, it is sufficient for the students to use the available materials from textbooks and journals for their assignments, patient care, case studies, etc. This can be explained by the materials being within the immediate reach of the students. Information technology is yet to be fully appreciated, with many students preferring to go the old way. Nevertheless, availability and access to software and personal computers remains a major issue, because even the faculty does not have easy access.
An important method of creating awareness is having the faculty require students do a MEDLINE search for information. It has been reported that connecting assignments and other academic activities to MEDLINE searching brought about an increase in the awareness and use of MEDLINE [2]. The organization of periodic training programs in the use of computerized information searching systems by the library and information-science staff would prove useful. For example, a short formal course of instruction in computer skills (computer basics, e-mail management, and MEDLINE and Internet search tools) organized for first-year medical students entering the State University at Stony Brook, New York, USA, was successful in achieving an acceptable level of comfort in using a computer for almost all of the student body [13].

Introduction
Of the three major pressures on health services worldwide, changing demography, growing expectations of the public, and new health care interventions (technologies), the last is generating the most concern among decision makers in health care and also the most dramatic responses [1].
Health technologies are the drugs, devices, and medical and surgical procedures used in health care, and the organisational and supportive systems within which such care is provided [2]. Thus, a cardiac monitor is a technology, and an intensive care unit is also a technology.
Ideally, the introduction of new health technologies is aimed at improving the health of patients, but not all new technologies bring about health improvement or show a reasonable balance between patient benefit and costs [3]. Historical cases like DES (diethylstilbestrol) (see textbox) and many other studies have shown that new health technologies sometimes spread rapidly in a health care system, even though there is no convincing evidence on safety and effectiveness [4]. The opposite situation, although less frequent, has also been documented, resulting in underuse of beneficial and cost-effective new technologies, e.g. laser treatment of diabetic eye disease [5].
These examples illustrate that decision making on the uptake and use of a health technology needs to be supported by high quality information. An important tool for decision makers is health technology assessment (HTA). This is defined as the analysis of the implications of a health technology in terms of its safety, efficiency, effectiveness, accessibility and equity, with the aim of supporting appropriate use of health technologies by providing input to decision-making in policy and practice [6]. Textbox 1. The case of DES Diethylstilbestrol (DES) is a synthetic female hormone first produced in 1938. It has a number of uses. This case focuses on the use of DES for complications of pregnancy. DES was approved for marketing in the United States in 1941. Several uncontrolled studies carried out by the advocates of the drug presented reduced pregnancy accidents. These studies led to its frequent use in pregnancy, and were used by the industry to actively promote the use in complicated pregnancies. During the period 1950-55 there were 7 controlled studies showing DES to be ineffective. Nonetheless, promotion continued, and in the 1960's DES was frequently used over much of the world. In 1970, a rare cancer of the vagina was noted in 7 young women. In all cases, their mothers had taken DES during pregnancy. DES then gradually fell out of use in both America and Europe.
DES was a case of a treatment that was both useless and harmful. It is a reminder that technologies should be proven beneficial before they are widely used.
HTA was developed in the United States in the `70's and has since gained momentum in Europe and the rest of the world. Besides concerns on the benefits and harms of health technologies, concerns on rising expenditures for health care were an important motivation for development of the field. Increasing health care costs can partly be attributed to the growing numbers of new health technologies. To provide an illustration, during the entire decade of the 1990s, 370 new drugs were brought to market in the U.S., an increase from 239 in the 1980s. The Pharmaceutical Research and Manufacturers of America (PhRMA) states that currently, pharmaceutical and biotechnology companies have well over a thousand new medicines in clinical trials or awaiting approval by the Food and Drug Administration (FDA) [7]. This rapid change and growth has also been witnessed in other types of health technologies in recent years.
Historically, most health technology assessments focused on technologies that were in relatively widespread use in the health care setting. However, when HTA of established technologies became common practice, the need of more timely assessments was recognized. In the nineties, it was increasingly recognized that being proactive, by identifying technologies before they were licensed or launched and by producing timely assessments of these technologies, would be advantageous. In general, early notice allows decision makers time to consider possible approaches to handling a new technology within a health care system [8]. To rationalize and manage this process of early notice, Banta and Gelijns [9] stated that a systematic approach is needed to identify emerging and new technologies, to select those that are important, to assess the consequences, and finally to disseminate this information to decision makers. The systematic handling of these steps constitutes the activities of so-called Horizon Scanning Systems (HSSs). In general, the purpose of a HSS is to help control and rationalize the adoption and diffusion of new technologies in health care practice, by providing policy makers timely information on the consequences of introduction of the health technology into the health care system.
The Netherlands was among the first countries to establish a HSS [10]. Nowadays, a number of countries worldwide have established a HSS. Most horizon scanning systems evolved from the work of HTA agencies in the nineties, the main difference being the focus of HSSs on technologies early in their life cycle. As a consequence, many HSSs are part of, or connected to HTA agencies.
The majority of these systems are members of The European Information Network on New and Changing Health Technologies, EuroScan , representing agencies in Canada, Denmark, France, Israel, Norway, Spain, Sweden, Switzerland, The Netherlands, and The United Kingdom [11]. All HSS in these countries are at least 50% funded from public sources, and target central and local health care policy makers with their early assessments. Furthermore, there are a number of organisations that provide the same services and use the same methods, but are not government funded. These include the Australian Safety and Efficacy Register of New Interventional Procedures-Surgical (ASERNIP-S) in Australia, and the University HealthSystem Consortium (UHC) in the United States.
In general, HSSs are interested in identifying potentially significant technologies for health and health care that might become available on the market in 0 to 5 years time. To identify these technologies, various sources have been recommended including consultation of individual manufacturers and clinicians, written sources such as pharmaceutical and medical journals, and the World Wide Web [1]. A combination of sources is recommended, as this provides corroboration, increases the likely accuracy of any predictions and increases the amount of useful information regarding a new technology [12].
Of the different types of sources, the Internet holds the promise of timely and efficient searching. However, although HSSs have begun to use the Internet as a source of information, anecdotal evidence suggests that its use varies and is generally unsystematic. As the efficient identification of new health care technologies is of the utmost importance for HSSs, this paper describes and analyses the current use of the Internet for this purpose by members of EuroScan and other selected horizon scanning agencies.

Methods
Drawing on a previous survey carried out internally within EuroScan, those horizon scanning agencies who use the Internet in one way or another in the operation of their system were identified. A questionnaire was sent to these agencies (n= 8) in April 2002. In addition, it was sent to two agencies outside the network that identify and assess emerging health technologies.
The questionnaire included 10 questions covering the following topics: • The scope of the scanning activity The first questions addressed what type of technologies the agencies scanned for, and which clinical specialties were priority areas for the scanning activity. Furthermore, agencies were asked which web sites they visited and were asked to appraise the selected sites to establish which sites were the most useful for them. The reason for these questions was to see if the agencies employed a strategy for which sites to scan based on type, importance of the site and the frequency of scanning. Finally, the agencies were consulted on how in general they prioritise web sites, and how much time they had available per week for scanning.

Results
All 10 agencies responded (100% response rate). At the time of the survey, three agencies (30%) indicated that they did not use the Internet to systematically identify new health technologies. These are the Health Council in the Netherlands, the Committee for Evaluation and Diffusion of Innovative Technologies (CEDIT) in France, and the Agencia de Evaluacion de Tecnologias Sanitarias (AETS) in Spain. In these agencies the Internet was used as a secondary source that is, to search for information on already identified technologies to enable them to prioritise these. The remaining agencies (n=7) all used the Internet as a primary source of information (see Table 1). Six of these completed the questionnaire.  Table 2 summarizes the responses of the agencies on which type of technology they look for (drugs or non-drugs), which specialty areas are included in the scanning activity, and if they scan different types of sites to identify different types of technology. Table 2 shows that most agencies scan for all types of health technologies. Two agencies, ASERNIP-S and SORTEK limit their scanning activity to identify new medical devices and procedures. In addition, all agencies, except ASERNIP-S and DACEHTA, focus their scanning activities on all specialty areas. These latter agencies focus on surgery and oncology, respectively. The final row in the table indicates if agencies scan different types of sites to identify drugs or medical devices and procedures. Four agencies say they do not, and two agencies say they do scan different sites for different types of technologies. Of the seven agencies that use the Internet as a primary source of information to identify new health technologies, six provided information on the frequency of scanning, the URL of sites scanned, and their relative importance. In total, the agencies scanned 110 different web sites. One agency, University HealthSystem Consortium, provided us with a list of 52 sites scanned, which have been described as part of a chapter in an electronic textbook on resources for Health Technology Assessment. This textbook is located at the web site of the National Library of Medicine [13].

Scope of scanning
The total number of sites scanned by the 6 agencies that provided us with complete information ranged from 11 to 27. The frequency of scanning is shown in Table 3.  Table 3 illustrates that most of the web sites are scanned weekly (41%) or monthly (33%). Two web sites provided listservs, that is, a service that sends selected information from the site to your personal e-mail address on a daily or weekly basis. For each agency, the number of web sites scanned daily ranged from 1 to 5 web sites, and the number scanned weekly ranged from 3 to 15 web sites.
Thirty-one percent of the total number of websites is considered highly important for the identification of new health technologies. Forty-three percent of sites were considered important, and 24% less important. Two sites were not evaluated, as there had been limited experience of scanning these sites (see Table 4).

Importance of sites
Six agencies provided information on the relative importance of individual web sites in their selection of routinely scanned sites. The web sites that were judged `highly important',`i mportant' and `less important' for the identification of new health technologies are listed in Table 5, Table 6, and Table 7 respectively. Out of a total of 110 sites judged, 16 (15%) sites were evaluated by more than one agency.

Succinct and Timely Evaluated Evidence Review (STEER)
The sites have been categorised into different types, according to their main features and purpose, such as sites containing regulatory information on drugs and devices for the US and Europe, information on specific types of technologies (drugs, devices, procedures), information on specific specialties, information from newswires, and information on new health technologies identified and/or evaluated by other agencies (see Table 5).
Of the sites containing regulatory information, the FDA (United States Food and Drug Administration) web sites are scanned by both European and North American agencies and are rated as highly important. The FDA provides a free e-mail service for news on both newly approved drugs and medical devices, FDA-NewsDigest-L (the Center for Devices and Radiological Health for devices, and the Centre for Drug Evaluation and Research for drugs), which users rated as highly important. In addition, the European Agency for the Evaluation of Medicinal Products (EMEA) web site provides information on approved drugs in the European Union, and outlines the evidence base for approval. On a commercial basis, the F-D-C reports' site allows access to the table of contents and brief summaries of information contained in F-D-C publications. Of these, the `Pink Sheet' covers the latest regulatory, legislative and business news affecting the US prescription pharmaceutical industry (http:/ /www.thepinksheet.com/FDC/Weekly/pink/TOC.htm). Thè Gray Sheet' focuses on medical devices, diagnostics and instrumentation (http://www.fdcreports.com/grayout.shtml).

Location/address of web site (URL) Source of information Type of information
One of the agencies, the University HealthSystems Consortium (UHC), further recommends the NDA pipeline compiled by F-D-C reports, a weekly updated database for tracking drug and biological product research, clinical trials and approvals. It is accessible on subscription basis only (http://www. ndapipeline.com/c3/welcome/welcome.plex).
For news on drugs in the pipeline, DrugInfoZone was used by two agencies. Most information on this site is password protected (for National Health Service staff in the UK only). The site includes, amongst others, a database of recent drug launches, a patents database, and drug reviews. Furthermore, it provides a free daily e-mail service with news from a number of sources including newslines such as Reuter's health, medical journals, pharmaceutical journals and other health-related web sites. The PharmaLive site was scanned by one agency. It is a web site targeted towards the pharmaceutical industry, providing news items on research, marketing, and regulation of drugs. It is a commercial web site. Access to more detailed pipeline information is on paid subscription basis only. The only web site rated as highly important that Horizontal Scanning Systems agencies identified as providing valuable information about medical devices is Medical Data International. However, this news service has recently introduced subscription fees. Reuters Health's web site is recommended for news on new health technologies in general. This web site is used as a primary source by many other health information web sites as well. Examples of specialty-specific web sites are SurgeryLinx, providing summaries and access to journal articles on new surgical procedures, and Doctor's Guide Haematonews, which lists news items on all types of cancer. Table 6 shows the sites that were valued as important by the responding agencies. Table 6 shows that, compared with Table 5, a new category of sites `Health portals' has come up. Health portals includes sites like Medscape and Doctor's Guide to the Net, that report on information on different types of health technologies from newswires, clinical conferences, and journals, and provide the possibility to search for this information in a great variety of predefined clinical specialties. The categories `Information on new drugs' and `Other EW or HTA organisations' have disappeared, because no sites were listed in these categories. Table 7 shows the sites that were valued as `less important' by the agencies.
Compared to Table 5 and Table 6,Table 7 shows two new types of sites: `Consumer Health Information' and `Journals'. The category of `Information on new drugs' does not exist in this listing of sites.  Table 8 presents an overview of the 16 sites that were scanned by more than one agency. Table 8 shows that half of the sites are only scanned by 2 agencies. In addition, in the vast majority of sites there is a discrepancy between the agencies with regard to their relative importance. For example, Medscape is scanned by 5 agencies, and is valued as highly important by one agency (indicated by 1), and as important site by the other four (indicated by 2). Medscape is a health portal, and provides information from newswires, journals, and conferences on a variety of clinical specialties. This information can be accessed on their home page, but also by specialty on specialty pages, for example Medscape Haematology-Oncology. Reuters Health is scanned by 4 agencies. This site is valued very differently, from highly important by one agency, important by another and less important by a further 2 agencies. In general, Reuters Health is considered a valuable source for news related to health and medicine. The site provides abstracts on news items that enable users to judge the item's value.

Overlap in sites scanned
The EuroScan web site is valued as either highly important or important by the four agencies that scan this site. The site is mostly visited by members of this information network. The site contains a database, only accessible to member agencies, which includes information from members on new health technologies, enabling exchange of information between the members.

Identification and prioritisation of new web sites
The responses to this question indicate that new web sites are mostly found through word of mouth (colleagues), or through links from one site to another site. Frequently, new sites are prioritised by an information specialist in an informal way. One agency's response was that sites are trialled for 1-2 months and that after that a recommendation is made to include or exclude the site in the routine scanning activity. In either method of prioritising sites, the same set of criteria is used to prioritise one site above another. The agencies responded that sites that appear to produce more or a similar amount of useful information than sites that are already scanned are likely to be added to the list of sites to scan. Furthermore, sites are most attractive when they are easy to scan, provide an e-mail service, are free of charge, and when they appear to provide objective information.

Available time to scan web sites
The responding agencies use at least 2 hours, and at most 8 hours per week to scan. Of the agencies, one scans less than 3 hours, 4 scan between 3-6 hours a week, and another up to 8 hours a week. One agency provided a range of 2-8 hours.

Discussion
Although the absolute number of agencies that have been covered by the survey is small, we have reasons to believe that it has covered most, if not all, HSS in the industrialized world. . Firstly, through cooperation with EuroScan it was possible to identify all member agencies that in one way or another use the Internet as part of their horizon scanning system. These agencies were complemented by two other agencies that carry out horizon scanning activities but who were not members of EuroScan. Because of the nature of the horizon scanning activity the vast majority of agencies in our sample is publicly funded, mostly by central governments. Agencies also tend to be relatively small, consisting of 1.5 to 7 full-time equivalents [14]. In addition, scanning is only one of the activities involved in the operation of most HSSs, with prioritisation and early assessments of new health technologies being other important functions. As a consequence of this resources for scanning are limited. Taking into consideration that the number of health technologies that emerge from pharmaceutical pipelines and manufacturers portfolios is great, the need for an efficient scanning strategy for EW agencies is evident.
The general picture that emerges from the results is that around half of the horizon scanning agencies actively uses the Internet as a source of information for identifying new health technologies. Furthermore, the agencies that have made this step on average spend considerable time on this activity. This is illustrated by the fact that the majority of the agencies scan for 3-6 hours a week, and that around 40% of selected sites are scanned weekly with some being scanned daily.
In total, the agencies scan a large number of sites (n=110). However, only 15% of these sites are scanned routinely by more than one agency. The relative lack of overlap in sites scanned can be partly explained by differences in scope of scanning between different agencies (e.g. only focusing on drugs versus including all technologies). Other factors that contribute to the diversity of sites being scanned and the lack of overlap of scanning activities between agencies include the great number of web sites available on the Internet, and the fact that sites are frequently selected in an unsystematic, informal way. Individual preferences of local information specialists may therefore be of paramount importance for the outcome of the selection process.
Similarly, differences between agencies in their rating of individual sites may occur due to the factors listed above. In this regard, the finding that 24% of the sites scanned were judged as less important has been surprising. One may wonder why agencies scan less important sites, when at the same time resources are limited? One explanation could be in what Wagner [13] defines as a preferred method of scanning, including the scanning of both `core' and `adjunct' sources (sites), that is first scanning those sites that have proved to yield most valuable information, and when there is time left, scanning additional sites that could yield supplementary information.
We feel that this subjective assessment by the agencies, of the relative importance of web sites that they scan, could serve as a starting point for discussion between agencies in order to arrive at common criteria to determine the usefulness and importance of web sites for identifying new and emerging health technologies.
Although a large number of sites are scanned (n=110), these can be categorised into a much smaller number of types of sites (n=10) that have a similar purpose. The categorization has been made on the basis of the main feature and purpose of a web site, and is as such not totally mutually exclusive. Of these categories, some are used more and are rated higher than others. Prominent types are `Regulatory information', `Information on new drugs', Specialty-specific sites', `Newswires', sites of `Other Horizon Scanning or Health Technology Assessment organisations', and possibly `Health Portals'. This might point to their importance to include in a search strategy for identification of new health technologies.
We conclude from the survey that there is marked variation between horizon scanning agencies in the way they use the Internet for identifying new health technologies. We have the impression that these differences can only partly be explained by differences between individual agencies in terms of e.g. source of funding, scope of scanning, and so forth. Factors that may be equally or even more important in explaining variation are that identification of technologies using the Internet is a rather new activity, and that so far there has only been limited exchange of information on this subject between agencies. We therefore recommend, given the resources used on scanning, that agencies become more selective in their choice of web sites and perhaps try to define a more transparent, operational distinction between highly important, important, and less important sites for the identification of new health technologies.
Horizon scanning agencies may benefit from further investigation into which sites deliver most output. Exchange of information between agencies about valuable sites, and a more formal selection process of new web sites on the basis of selected criteria could result in a more efficient scanning process. A future activity could include a coordinated effort to develop Internet scanning strategies for different categories of health technologies or different clinical specialties. This may improve efficiency and quality of scanning in terms of numbers of potentially relevant technologies identified.
In practice, the Internet does not stand alone as a source. Most agencies use a combination of sources, such as information from clinical experts and manufacturers, scientific journals, grey literature, and conference material. It is therefore furthermore recommended that future Internet scanning strategies fit into the broader search strategy of agencies.

Introduction
The Internet has become increasingly popular among consumers as a source of health care information. A poll in 2001 showed that almost 100 million American adults regularly go online for health care information [1].
Although there are quality-standards guidelines for medical publishing on the Internet [2,3] there is currently no governing body acting as a gatekeeper of Web page publications. Anybody can create a Web site and they are free to write whatever they wish. This has led to some serious concern as to the accuracy of the information of some of these sites [4,5]. Even those sites that offer to evaluate health Web sites are often incomplete or fail to reveal how they perform the evaluation [6,7].
LASIK (Laser in Situ Keratomileusis) is a very popular [8] combined surgical and laser procedure, which is used to correct myopia (shortsightedness) and hyperopia (farsightedness). As the procedure is not available within the United Kingdom's National Health System, but is available privately, many commercial companies have taken a keen interest in promoting its uptake. Although LASIK is predominantly a safe procedure there are many potential complications [9][10][11][12][13][14][15][16]. Most of these are mild and/or transient but some are severe, permanent, and may require corneal grafts to correct [10,11]. LASIK was first performed only 11 years ago and United States Food and Drug Administration approval was only granted 6 years ago; therefore there is concern as to what may happen to these corneas in the future.
Based on a Medline search there have been no previous studies that have specifically examined the quality of ophthalmic Web sites. This study examines information a member of the public might read when searching for information on LASIK on the Internet. The sites are evaluated regarding the quality of the content, authorship, and technical aspects of the information on the complications of LASIK. that have specifically examined the quality of ophthalmic Web sites. This study examines information a member of the public might read when searching for information on LASIK on the Internet. The sites are evaluated regarding the quality of the content, authorship, and technical aspects of the information on the complications of LASIK.

Methods
Between July 16 and July 22, 2002, 21 Web sites that described LASIK were evaluated. Included in the evaluation were the first 17 English-language sites that appeared when LASIK was searched using the Google search engine [17] and the first 4 UK (United Kingdom) sites selected using the same keyword but using the Yahoo [18] search engine and limiting the selection to UK sites. Although the Google search returned 150000 hits and the Yahoo search returned 1010 hits, the evaluation was restricted to the first 21 English-language sites that appeared, as it was felt that most English-speaking potential LASIK customers would not extend their search beyond this number of sites. There was no consumer involvement when devising the search strategy. Of the Web sites included in the study, 16 were North American, 4 were United Kingdom, and 1 was Indian. Web sites that just provided information on addresses of LASIK surgeons/surgeries, were not included.
Each site was assessed by one rater (DF) who was blinded to the source. Objective measurement was performed of the following: source. Objective measurement was performed of the following: • Whether the Web site was commercial, academic, government, or by an individual. an individual.
• Where the site was from (ie, United States, United Kingdom, or elsewhere). elsewhere).
• Whether the site dealt solely with LASIK, with different types of eye laser procedures, or with various eye conditions. of eye laser procedures, or with various eye conditions.
• Whether information on complications was given.
• Whether a "last-updated" record of the page with the information on complications was given."last-updated" record of the page with the information on complications was given.
• Whether the author of the information was identified.
• Whether links (ie, to the explanation of medical terms) or relevant graphics were used to explain the information. relevant graphics were used to explain the information.
• Whether responsibility for the information given is waivered. waivered.
• Whether a consent form is available online.
Most of the information on complications was assessed and marked subjectively based on authorship, content, and technical quality. The evaluation form used (Table 1)

Recognised Authority
This score was based on whether and to what degree the author is a recognized authority in the area. The sites scored higher if the authors were ophthalmologists, if they perform many LASIK procedures themselves, and if they have publications in the area. a recognized authority in the area. The sites scored higher if the authors were ophthalmologists, if they perform many LASIK procedures themselves, and if they have publications in the area.

Credentials/Experience
This score was based on whether and to what extent the author's experience and credentials were provided. The more information, given the higher the score. experience and credentials were provided. The more information, given the higher the score.

Contact Information
This score was based on the amount of contact information that was given to get in touch with the author for further enquiries. Full marks were given if a telephone number, e-mail address, and postal address were provided. was given to get in touch with the author for further enquiries. Full marks were given if a telephone number, e-mail address, and postal address were provided.

Details of Complications
This score was based on how much was written on complications. The more complications that were mentioned, and the more information (such as frequency, clinical course, and treatment options) that was given on each complication, the higher the mark. The more complications that were mentioned, and the more information (such as frequency, clinical course, and treatment options) that was given on each complication, the higher the mark.

Easy to understand
This score was based on how easy it would be for a member of the public to understand the information on complications. The highest marks were given when the authors felt the complications were well explained with diagrams and/or video and when medical jargon was either not used or well explained (either directly or by using hyperlinks). The information given was also analyzed objectively using the Flesch-Kincaid reading scale and Flesch reading ease score. public to understand the information on complications. The highest marks were given when the authors felt the complications were well explained with diagrams and/or video and when medical jargon was either not used or well explained (either directly or by using hyperlinks). The information given was also analyzed objectively using the Flesch-Kincaid reading scale and Flesch reading ease score.

Ease of Locating Complications
This score was based on how easily a member of the public could locate information on complications from the home page of the Web site. High scores were given when there was an obvious link to the complications from the home page, when the information on complications came under the heading Complications, and if there was a search engine on the site that could target the complication information if the words risk or complications were input.

Accuracy of References
This score was based on the accuracy of the references given on the information on complications. All references were read (full text) and the highest scores were given if the referenced information was to be found there and if it had been quoted in context. the information on complications. All references were read (full text) and the highest scores were given if the referenced information was to be found there and if it had been quoted in context.

Up-to-date Information
This score was based on the degree to which the information was up-to-date. Highest scores were given for information from the most recent studies. up-to-date. Highest scores were given for information from the most recent studies.

Balanced Information
This score was based on the proportion of, and the degree to which, the information was balanced. Highest scores were given for impartial information that wasn't overly optimistic or pessimistic. which, the information was balanced. Highest scores were given for impartial information that wasn't overly optimistic or pessimistic.

Quality of Referenced Page's Title, Body, and Footer Footer
This score was based on the quality of the page with information on complications. Highest marks were given if the title, body of text, and footer were easy to identify and if they provided relevant information. on complications. Highest marks were given if the title, body of text, and footer were easy to identify and if they provided relevant information.

Ease of Identifying Web Site Headings and Subheadings Subheadings
This score was based on the quality of the Web sites headings and subheadings. Highest scores were provided when the Web site had clear intuitive headings and subheadings that the authors felt would aid the visitor's navigation through the Web site. and subheadings. Highest scores were provided when the Web site had clear intuitive headings and subheadings that the authors felt would aid the visitor's navigation through the Web site.

Statistics
Nominal data was expressed as percentages. Sample proportions were compared by hypothesis testing with a 5% significance level using the MINITAB statistical software package. Subjective scores for the Web sites were expressed as a mean and standard deviation, which also was calculated using MINITAB. were compared by hypothesis testing with a 5% significance level using the MINITAB statistical software package. Subjective scores for the Web sites were expressed as a mean and standard deviation, which also was calculated using MINITAB.

Results
General information about the Web sites visited is summarized in Figure 1 and Figure 2. 16 of the 21 sites visited were from the United States and 13 of the sites were devoted to LASIK only.  Figure 3 shows that 16 of the 21 the sites mentioned the complications of LASIK and this was more common in the American sites compared with UK sites (75% vs 50%), although only 4 of the sites visited were UK. This is not a statistically significant difference ( P= .285, 95% CI). Figure 4 illustrates that 12 out of the 16 US (United States) sites visited were commercial compared with all 4 of the UK sites. This is not a statistically significant difference ( P= .264, 95% CI). Twelve out of the 16 commercial sites compared with 4 out of the 5 individual/government sites gave information on complications of LASIK, which is also not a statistically significant difference ( P= .819, 95% CI).   Table 2 shows that out of the 16 sites that had details on complications, 2 showed when they were last updated, authorship of page could only be clearly identified in 3, and referenced information was only found in 6. Of the 16 sites visited only 2 used links to enhance the quality of their complication information and only 6 waivered any responsibility for the information given. Figure 3 showed that 5 of the 21 Web sites visited had no information on the possible complications of LASIK. In those sites that have information on complications, Table 2 and Table  3 illustrate that it is often inadequate because of: • Poorly identified and/or inappropriate authorship. • Not up-to-date.
• Imbalance. Table 3 shows that of the 2 Web sites that provided information on the complications the references were generally accurate (score 8 out of 10) and were mostly were up-to-date (score 6 out of 10). One of these Web sites scored 2 out of 10 for balanced information because there was only 1 quoted reference for all its information; this was from a commercial site.
Although this study presents a mostly-negative picture of the complication information on LASIK Web sites there were some positive findings. Table 3 shows that most sites scored well on the quality of headings, subheadings and footers.

Discussion
This study assessed the quality of information on the Internet on LASIK complications and found it to be poor. Our findings support recent work in the College of Optometry in Southern California [22], which looked at 96 Web sites containing LASIK information. They rated 26% of sites as "markedly informative," 28% were rated "moderately informative," and 46% were rated as "minimally informative." The poor quality of the information represents a negligent omission as the public are being misled into believing that LASIK is without risk. This may lead to liability cases by patients with complications whose decision to have LASIK was based on the information they read on the Web site. Of the sites that gave information on the complications of LASIK, only 38% (Table 2) included a waiver of responsibility for the information. There is an argument that the authors of the Web site could be held liable unless they have a waiver. On the other hand perhaps health Web sites should be prevented from using a waiver to ensure greater accuracy of the information provided.
Detailed, well-referenced, up-to-date, good quality information from recognized authorities should be found on all medical Web sites. This is particularly important for procedures such as LASIK that are often primarily cosmetic but whose complications can be devastating. The risk of irreparable sight-threatening complications such as corneal ectasia [11] particularly needs to be explained. The combination of graphics, video, and hyperlinks should be fully utilized to explain difficult concepts and allow patients to make fully-informed decisions regarding their medical treatment. Comprehensive consent forms available on the Internet should be mandatory for elective procedures such as LASIK. This was only seen in 1 of the Web sites analyzed [23].
In a systematic review of studies that assess the quality of health information for consumers on the World Wide Web, Eysenbach et al [24] conclude that "the individual's risk of encountering an inadequate site on the Web is a function of both the proportion inadequate information on the Web (P) and the inability (I) of the individual (or his tools) to filter the inadequate sites." Because of the large public interest in LASIK it is the opinion of the authors that all LASIK Web sites should adhere to strict standards regarding what is published. The public is more likely to trust information from a government/educational site, but all the UK sites located in our study were commercial. It suggests that the UK government and/or ophthalmology institutions need to produce a good quality Web site on LASIK for the public to use.
Deciding on the most appropriate method used to control/guide health care information on the Web is difficult. Current methods include filtering tools, quality labels, codes of conduct, and user-guidance labels [25]. The National Electronic Library for Health uses "methodical, organized human reviewing, selection and filtering based on well-defined quality appraisal criteria" for their site [26]. There is a subjective bias in this system and one might argue that filtering systems violate human rights by acting as a censor of health information. The eHealth Code of Ethics of the Internet Health Coalition [27] is an example of a code of conduct. The coalition is an organization that has developed a set of quality criteria for those wishing to produce health-related Web sites. If the public knows that a Web site adheres to a certain standard when it was produced then one could avoid misleading Web sites that adopt misleading tactics such as picking up the client machine's current date and displaying it after the "last updated" remark, thus giving a false impression of the currency of the information [26]. Net Scoring [28] is an example of a user guidance system that enables users to check if a site and its contents comply with certain standards by accessing a series of questions from a displayed logo. This requires a considerable amount of effort on the part of the user. It is the opinion of the authors that the best solution is to have accredited third-party approval "stamps" that would be clearly visible on the Web sites. For example, the LASIK Web sites might have a comment after the description of the complications that says "This information has been reviewed and approved by the Royal College of Ophthalmologists, UK." Selection of a correct approach is controversial and some authors such as Delamothe [29] argue that "rating the quality of medical Web sites may be impossible " and that "one option is to rate the process by which the content was produced rather than the content itself -a medical journal's Web site containing peer-reviewed material would rate higher than a commercial site selling miracle cures for cancer." The results of this study are somewhat limited in that part of the evaluation is subjective; however, this is balanced with some important objective assessments (such as the number of sites which mention complications). the evaluation is subjective; however, this is balanced with some important objective assessments (such as the number of sites which mention complications).
Many LASIK Web sites are giving minimal, dated, poor-quality, and inaccurate information on its complications and this can mislead the public. In many cases there is no authorship of the information and in those cases where authorship is clear the information may not come from a recognized authority. Authors should publish information that is easy to understand and locate. The explanations of the complications should be supplemented by the excellent multimedia capacity of Web sites with many images and links. One solution is for authorities such as the Royal College of Ophthalmologists to evaluate UK ophthalmic sites and offer to provide a stamp of approval for those that fit their good-quality criteria. We conclude that the quality of information on the Internet on the complications of LASIK is poor and needs to be addressed. As Wyatt [4] says, "unless we evaluate the quality of clinical sites and their effects on users, we risk drowning in a sea of poor quality information."

Introduction
The Internet has unquestionably become an important aspect in many facets of our daily lives. In the particular area of education, many useful exploitations of the Internet have been in existence for a long time, with courses in many fields being offered online [3].
In the medical physics community, the Internet has already assumed an almost indispensable role, with the dissemination of information and the facilitation of communication [4,5] as well as electronic journals with extensive literature search capabilities, and more recently the cataloging of virtual online seminars from conferences as well as online courses, such as those offered by the American Association of Physicists in Medicine (AAPM) [1].
One aspect that hitherto has not received much attention within the medical physics community is the area of real-time interactive remote education using the Internet, and this is the subject of our present study. This form of real-time education would resemble the common remote classroom-type education, where the video and audio signals of the lecturer are transmitted to a remote classroom and shown on a television set, and the class can pose questions back to the lecturer.
Many factors would favor such a mode of teaching to the medical physics community. Medical physics is a relatively small community, with instructors and students often at great distances apart, and usually with a scarcity of expert teachers. Medical physics services are often lacking but needed in remote areas, including developing countries, and bringing access to expertise to these places would be beneficial. Medical physics also involves many software packages, concepts, and ideas that can only be best taught in a real-time interactive fashion, as well as ideas and information that are time-critical, such as a new protocol or new method of treatment or planning. In addition, a flexible course format allows a particular course to be tailored specifically for a small group of recipients. A simple communication method would also enable experts in the field to provide lectures easily from a convenient location, without the necessity for complicated computer system set-up and support.
The emphasis of the present project then is to investigate the format of real-time remote education that can draw the maximum benefit out of the Internet and benefit the community; in particular aiming towards the medical physics community in developing countries.

Methods
A collaboration was set up between the Department of Medical Physics at the Toronto-Sunnybrook Regional Cancer Centre in Toronto, Canada and the Department of Radiology at the University of Malaya in Kuala Lumpur, Malaysia, whereby a class of medical physics graduate students at the University of Malaya attended lectures provided by lecturers in Toronto, using the Internet as the main tool of communication. There are two aspects of the project. The first is to experiment with the method with which the real-time interactive communication is provided, as well as the logistics of communication (such as the time-zone difference and availability of the facilities). The second aspect is to determine the optimal contents and formats of such a remote-education project.
A survey of various tools available for tele-education was undertaken. These are summarized in the Appendix and described briefly below. The main "hardware" tools used include a personal computer (PC) at both the instructor's and students' end, with both systems connected to the Internet. In particular, the instructor's computer is connected via a high-speed connection to a local ISP (Internet Service Provider), and the students' computer was connected to the Internet through the university's computer network. Other hardware (such as a regular telephone connection, with a speakerphone) as well as other personal computer accessories (such as a Web camera, microphone and speaker sets, and a drawing tablet) are additional optional hardware that are listed in the Appendix. The software tools include a screen-sharing program by which the class can view the computer screen of the instructor, and hence the lectures could be provided using, for example, PowerPoint slides. The screen-sharing program could also allow the class to interact directly by typing or by drawing on a whiteboard, as well as -with the permission of the instructor -by directly controlling the mouse of the instructor's computer to advance slides or run other programs. Another useful program is one that can provide audio/video communication between the two sides. An alternative to that is to use the regular telephone for audio communication only. As part of the study, many of these tools were tested to determine the usefulness and limitations. The observations are summarized in the Results section.
The curriculum of the course compiles a series of lectures with various topics in radiation therapy. The class consisted of 7 medical physics graduate students who are enrolled in a regular medical physics program at the University of Malaya that covers both imaging and therapy subjects. The topics for the tele-education course were chosen to supplement the regular program, with special attention paid to hands-on demonstration of software packages and to topics requiring more interactive communication between the two sides.

Results
A series of one-hour lectures, including demonstration of software, were given during the course of the project. Figure 1 shows a scenario of the lecture, where the instructor's PC screen appears on the screen of the students' PC, together with the video picture of both parties using the software program NetMeeting [2].
A typical scenario of a lecture proceeded as follows. The details of the lecture, such as the time, the topic, the mode of audio/video communication, would be set up ahead of time via e-mail. At the time of the actual lecture, the instructor would first establish voice communication with the class, via a regular telephone call. The class would then use their personal computer to access the Instructor's computer via the Internet. Once computer communication was established, the class could view the PowerPoint slides that the instructor brought up on his workstation. The lecture would then proceed using the regular telephone network through a speakerphone, as well as the screen-sharing program. The class can interact via the speakerphone as well as the screen-sharing program.
Since this is a pilot project, the students were asked to provide feedback on all aspects of the project. Because of the relatively small number of students involved, the quantitative aspects of the evaluation are not meaningful, but the overall evaluation can be summarized below.

Time Difference
One of the major difficulties was the 13-hour time difference between the two locations. This greatly affected the testing of the hardware and software tools, as well as the organizational aspect of the actual lectures. It was most practical for the instructor to deliver the lecture from home in the evening, using a laptop computer connected to a high-speed Internet service provider. The students would then attend the class at the university during the morning. The times had to be reversed for some of the workshops where the software was not available on the laptop computer, and so the instructor demonstrated the software at the office in the morning, while the students held the class at the University in the evening. Once the students were familiar with the software, they were allowed to log into the instructor's office computer using a guest account and run the software program independently. The students also managed to download the PowerPoint slides for offline viewing.

Communication Method
The preparation for the actual lectures involved experimentation with and the analysis of the different communication facilities and methods. These are summarized in the Appendix. The configuration we found most reliable in carrying out the series of lectures is:

Availability of the Internet
The speed of the response depended critically on the traffic on the Internet network, which could vary significantly with the time of day. When the lectures were given in the evening in Malaysia the response was acceptable, with a screen refresh usually transmitted to the students' computer within 10 seconds.

Screen-sharing Software
The screen-sharing system GoToMyPC [6] was used as the tool for the lecture presentation. This system requires a subscription to the service provider, after which a series of instructions on the Web site allows the easy installation of a program on the host computer. The students then go to the same Web site and follow some simple instructions to access the instructor's computer. This software was the most suitable one for our requirements because of its ease of use, its ability to work properly in the presence of firewalls, and its relatively low cost.

Audio Communication
Effective audio communication was critical to the success of the lectures. Using the regular telephone network was most reliable, and the cost was relatively low.

Video Communication
The video communication using the computer software NetMeeting as depicted in Figure 1 was also attempted but because of the firewall problem was not successful; but is still being pursued. The lectures managed quite well without video communication.

Firewall
The presence of the firewall proved to be a major hurdle that prevented the use of many software tools. The simple combination of GoToMyPC and the telephone proved to be sufficient to allow us to complete a series of one-hour lectures.

Course Contents
A standard medical physics curriculum already exists at the medical physics department at the University of Malaya, so a series of special topics, such as IMRT (Intensity Modulated Radiation Therapy), was selected for the lectures. A demonstration of a treatment-planning system was also carried out. It was found that lectures that can be given using PowerPoint slides were most suitable for this tele-education project.

Class and Instructor Interest
The class welcomed the opportunity to interact directly with experts in the field. The relatively-small community of medical physics would favor small class sizes and interactions, as well as the involvement of experts at distant locations.

Cost
It is encouraging to report that the cost of the remote-education program could be kept to quite a bargain-level. This, of course, depends on the availability of fast Internet access. The software program we used, for example, costs about US $100 per year to install on the instructor's computer. The other costs such as telephone costs and minor hardware items, are insignificant compared to, for example, travel costs to workshops. The major cost that remains would be the lecturer's fees. Since the process offers the lecturer a great deal of flexibility in terms of time and place for the lectures, the "cost" to the lecturer is kept to a minimum. In addition, this is a new and exciting process that many will find interesting to participate in. Hopefully this will encourage lecturers to donate their time for an affordable and worthwhile project.
To start a real-time interactive remote education program, however, careful planning must be undertaken. In a recent case study on using the Internet to teach health informatics [7], various technical difficulties, in particular for synchronous communication, were discussed. Computer and network problems could be very frustrating and minor glitches could discourage both the instructors and students. It is highly recommended that a new program be carried out with a minimal but reliable set of tools, such as the standard telephone used in our project, and preferably with the support of knowledgeable computer consultants.
The issue of whether the Internet is a net disservice to the medical physics community in developing countries is a recent debate topic [8]. Hopefully with the increased availability of high-speed Internet service, a remote-education program like the one reported here can bring practical benefits to the developing countries.

Video Communication
Although the lectures managed quite well without video communication, it is felt that its addition would have provided a valuable feedback mechanism.

Firewall
Although the simple combination of GoToMyPC and the telephone proved to be sufficient, it is felt that if a connection could be established without the blockage of the firewall then the method of communication could be more flexible and powerful. Because of this, it is recommended that the computer department in the organization or the university be involved if a firewall exists.

Instructors
Instructors could give lectures at any convenient location where there is access to the Internet, and the screen-sharing software used for the project could be uninstalled from one PC and reinstalled on the guest-lecturer's PC easily and legally.

Course Contents
Topics such as dose calculation would require more blackboard-type illustrations and are more difficult to carry out, although if a drawing tablet were available the process would have been easier. A particular advantage of this mode of tele-education is that very-specialized expertise could easily become accessible in otherwise impractical situations.

Conclusions
In conclusion we feel that the project proves that real-time interactive remote education of medical physics is a viable concept that is ready to be carried out with selected places and groups. It serves a useful purpose and is a very cost-effective way to promote closer communication and to spread knowledge and information within the medical physics community. There are certainly limitations and any new programs must be carefully planned. The increased usage of the method will speedily eliminate some of the existing limitations.

Abstract
This paper reviews trends and issues in health and in the information and communication technologies (ICT) market as they relate to the deployment of eHealth solutions in Latin America and the Caribbean. Heretofore designed for industrialized countries and large organizations, eHealth solutions are being proposed as an answer to a variety of health-system management problems and health care demands faced by all health organizations including those in developing societies. Particularly, eHealth is seen as especially useful in the operational support of the new health care models being implemented in many countries. The authors examine those developments vis-à-vis the characteristics of the Latin American and the Caribbean health-sector organizational preparedness and technological infrastructure, and propose policy and organizational actions to foster the development of eHealth solutions in the region.

The Foundation and Practice of eHealth
Advances in ICT and the dissemination of networked data processing created a new environment of universal access to information and the globalization of communications, businesses, and services. In the health sector, this trend is exemplified by the growing consolidation of eHealth -an area distinguished by the combined utilization of electronic communication and information technology to transmit, store, and retrieve digital data for clinical, educational, and administrative purposes, both at the local site and at distant sites. Barely in use before 1999, the concept of eHealth evolved to serve as a general designation to characterize not only networked health applications, but also virtually everything related to computers and medicine. It followed a trend started by other "e-words," such as e-Commerce, e-Business, e-Finance, e-Learning, e-Government, e-Solutions, and e-Strategies [1][2][3]. Among leading digital technologies, Internet-based ICT solutions are rapidly changing the way health providers, health plans, organizations, payers, regulators, and consumers access information, acquire health products and services, deliver care, and communicate with each other [4,5].
Most eHealth solutions build on e-Commerce and e-Government strategies and experiences in using Internet-based networked technologies to rethink, redesign, and rework how businesses and public services operate. Typically, such developments have been aimed at the improvement of productivity, effectiveness, and efficiency, both internally and in the relationships with clients, customers, suppliers, and partners. Reaping the full benefits of such innovative data processing and use depends on the clear definition of goals, collaboration among stakeholders, technology infrastructure, systems integration, standards, and the implementation of performance metrics.
The business imperative for eHealth is concrete, is driven by the operational requirements of health reforms, and is aligned to many of the determinants found to be relevant in e-Commerce [6][7][8]: • Growth of a global marketplace and the ubiquity of interactive communications.
• Networks of producers, suppliers, customers, and clients.
• Global demand for telehealth services is estimated to be of US $1.25 trillion, of which about two-thirds is for direct services and the rest for second opinion, consumer information, continuing education, management, and other services.
• Leasing, membership, service agreement, and strategic-alliance models replace traditional business organizations based on ownership of physical assets and long-term structures.
• Lifetime value of customer retention replaces "one-time sell." • Economies of speed, forecasting demand, and customer service and satisfaction replace economies of scale and impersonal service provision.
• Customization capable of achieving a "one-of-a-kind" product or service.
• Leveling effect by reducing entry barriers, thus allowing small firms and poor countries and populations to have access to markets, information, and other resources, and therefore balancing the vertical integration competitive advantage of large corporations.
The essence of eHealth, as in e-Commerce, is reliable transaction delivery in a fast-changing environment involving people, processes, and a business infrastructure focused on the ill or healthy citizen. In developed countries, eHealth has rapidly evolved from the delivery of online medical content toward the adaptation of generic e-Commerce solutions to the processing of health-related administrative transactions and logistical support of clinical tasks. Emerging eHealth applications are oriented to professional networking, integration of the clinical care process management, and the provision of Web-based health information and patient care, including remote monitoring and health care. This expanded view of eHealth has been promoted as the final stage in bringing the entire health care industry online.

Organization and Delivery of Health Care in Latin America and the Caribbean
The health sector faces two demands that appear, on first examination, to be contradictory: firstly, to provide expanded and equitable access to quality health care services and, secondly, to reduce or at least control the rising costs of health care services. Although the health sector is key to the welfare of the population and the formation of human capital, the sector has not kept pace with the momentum of change that the region has experienced in recent years in other areas of economic, political, and social life.
Health sector expenditures in Latin America and the Caribbean comprise 6% to 17% of the service sector that, in turn accounts for 50% to 65% of the GDP (Gross Domestic Product) in almost all countries. The market for health goods and services in the countries of the region represent 9% of the global health market, about half of that for the European Union, above that of the combined markets of Eastern Europe and Central Asia, and just below that of East Asia and the Pacific. The average per capita expenditure on health in 1999 was US $452 followed by US $1868 for the European Union, US $2206 for Canada, and US $3978 for the United States. (PPP dollars). PPP (Purchase Power Parity) relates to the purchase value of the dollar at the local market. PPP rates allow a standard comparison of real price levels between countries. The PPP for a specific country is based on market price surveys conducted by the International Comparison Program, a joint project of the World Bank and the regional economic commissions of the United Nations. The present values of PPP are based on the last survey done in 1996 and is based on a 1993 reference year.
There is a marked variation (Figure 1) in the national expenditures among Latin American and the Caribbean countries even for countries of comparable income level [9]. Changing demographics (particularly age structures) and lifestyles (mainly due to epidemiological profiles, urbanization, and growing industrialization) highlight the need to reorient care models. In high-income and middle-income countries, about 40% of the population has one or more chronic conditions and, in many societies, chronic conditions account for up to two-thirds of health care expenditures. In each care setting, a limited set of health conditions account for most of the cost due to the growing demand on the care system for additional and high-cost diagnostic and therapeutic resources. Significant opportunities do exist to improve health status but there are still a considerable number of preventable diseases and premature deaths, both in absolute and relative terms, and there is great inequity of access to basic health services resulting in regions, communities, and social groups being left without access to the most basic health care.
In most countries, the health sector is underfinanced; this has led to quantitative and qualitative deficiencies in the delivery of health services and to growing gaps in basic care. There is inefficient allocation of scarce resources and lack of coordination between health subsectors, institutions, and other social agents and stakeholders -with duplication of efforts, overlapping responsibilities, and resource wastage. All countries are in some stage of sector reform, a process aimed at introducing substantive changes in the health sector and in the relationships among stakeholders and the roles they perform, with a view to increasing equity in benefits, efficiency in management, and effectiveness in satisfying the health needs and expectations of the population [10].
Health reform processes have many facets and there is no single model being adopted by all countries. Each country is moving at a different pace in the implementation of its own particular health system model but the economic and globalization changes of the last 10 years have brought a new urgency to the reform processes. There are, however, common trends and responses that characterize most health sector reform processes occurring in the region: (a) the universalization of a high cost-benefit basic package of health services; (b) a set of standardized public health interventions; (c) cost containment and recovery; (d) administrative decentralization and operation of health care services; (e) recognition of the role of the private subsector and the intersectorality of health interventions (involvement of other sectors -for example, education, environment, and laborbesides health in the determinants of health status and health care activities); (f) health models oriented towards primary care and centered on people; (g) focus on quality and accountability; and (h) moving away from the reactive delivery of care to a more proactive management approach of the health status of individuals and population groups [10,11]. With the introduction of ICT, health sector users (clients) and providers (professionals and organizations) will be forced to assume new roles (different from the traditional ones). Furthermore, the introduction of ICT will bring into the health sector environment a host of new professionals (eg, systems professionals) and will require novel forms of interaction with municipal and provincial (state) health authorities. Competition, merger of provider organizations, aggressive contracting by payers, and increasing involvement of employer and government purchasers have characterized changing processes in health services management. Information systems are essential for operational support and management of the new health and health care models [11][12][13][14][15]. They must address the needs of the new trends in health care that emphasize a continuous relationship between providers and clients; customization of care; expanding partnering of providers, insurers, and clients; increasing client control of evidence-based health decisions; information that is not frozen in records and kept in separate sites with access limited to their creators but available to all stakeholders; and transparency and cooperation instead of independent professional roles.

Socioeconomic and Development Constraints
Technology distribution and access problems represent the most acute issue in the dissemination of ICT applications. In a more limited focus, the "digital divide" encapsulates the dramatic worldwide variation in access to computer-based information technologies, usually more narrowly in terms of levels of Internet access, available to individuals and communities. Information-technology utilization inequalities are found in both industrialized and developing countries and are determined by level of education and income (Figure 2 Digital divides, like social and economic divides, exist within and not just between societies and are integral parts of a much broader and intractable "development divide" that include insufficient telecommunications infrastructure, high telecommunications tariffs, inappropriate or weak policies, organizational inefficiency, lack of locally-created content, and uneven ability to derive economic and social benefits from information-intensive activities [16][17][18]. The situation of technology adoption within developing countries has been one of growing polarization, with segments of the population bypassed by the products of the information revolution. This is complicated by the fast-changing deployment of new technologies and accompanying standards that are constantly raising the level of advancement that must be met by anyone who wants to remain current [19,20].

Technology Infrastructure, Market Determinants, and Operational Issues
In the health sector, development and digital divides between industrialized and Latin American countries are wider than the gap observed in other productive and social sectors. In some cases, the changes brought about by the privatization of health care did add to the already high degree of structural inequity that prevails in the countries of the region. Besides achieving reliable transaction delivery, a technologically successful "e-Architecture" must provide superior client service, customization of products and services, interactivity, and maximum convenience [21].  The deployment and operation of "e-Solutions" share technology-infrastructure and operational-deployment issues involving reliability of service that directly depend on: (a) degree of information preparedness and information technology insertion in society; (b) appropriate and functioning network, hardware and software platforms, and physical infrastructure; (c) the understanding of market relationships among the different actors in the informatics and telecommunications areas; (d) managing knowledge about health, individual client medical history, environment, and enterprises; (e) data-protection measures and regulatory framework to ensure transaction security; and (f) auditing processes that are quite different from traditional paper-trail solutions. A pervasive public-sector information infrastructure ("infostructure"), the essential prerequisite for continuous health care to the community, is still an incipient component of health systems. Penetration of information systems in health institutions is low. The hospital subsector is the area better served by information systems. Table  1 summarizes the distribution of computerized systems in hospitals.
Considering all facilities, public hospitals, including those belonging to the social security, account for 44.51%; private total 46.98%; philanthropic total 7.75%; and military the remaining 0.75%. There are significant differences in the utilization of computerized information systems. The relative distribution of information systems shows that social-security hospital facilities constitute only 5.29% of all establishments, but 50% have computerized information systems, followed by philanthropic (39.3%), private (36.7%), military (23.2%), and public non-social security (21.5%). However, in the group of all hospitals with computerized systems, private hospitals represent the majority (54.7%). The disparity between the existence of systems in the two types of public hospitals (public social security and public non-social security) is evident. In the past decade, the information technology sector in Latin America and the Caribbean consistently showed a growth of 17%, above all other world regions. During the last 5 years there has been fast expansion of the telecommunications market (from 7 million fixed-line and 50 million mobile subscribers in 1995 to 25.3 million and 69 million subscribers respectively in the year 2000) and very fast diffusion of the Internet (in 3 countries alone -Brazil, Mexico, and Argentina -the total number of registered geographic domain hosts grew from 326000 in July 1998 to 660000 in July 1999). Investments in ICT as percentage of the GDP is comparable to that of developed countries although the absolute value per capita is low (Table 3). Data used in the preparation of Table 3 come from sources that in one case use GNP (Gross National Product) and in the other case use GDP. GNP is the broadest measure of national income and measures total value added from domestic and foreign sources claimed by residents. GNP comprises GDP plus net receipts or primary income from nonresident sources Low penetration of telephony, averaging 12% in the region and cost of annual subscription (averaging 3% to 4% of the GNP per capita, but in some countries as high as 19.6%), and very low ownership of personal computers (2 to 10 for each 100 persons, average 3%), and low Internet connectivity (average 3%) are major challenges to be overcome (Table 4). About 2 out of 3 public ICT projects are deemed to be failures -they take a long time to implement, cost more than planned, and deliver less than planned. Most of the problems are related to the process of bidding, selection, and contracting.
Poor telecommunications infrastructure, limited number of Internet Service Providers (ISPs), lack of access to international bandwidth, and lack of affordable Internet-access costs are readiness issues that continue to be major impediments to diffusion of Internet applications to the point of care in developing societies. Dependable connectivity is needed for reliable transactions. Fast connectivity is still limited and is usually by dial-up access. A study across different industries showed that only about one-third of the connected organizations in the region had access with speed higher than 56 Kbps (Kilobits per second) ( Table 5). The access-site problem can be further illustrated by the result of a 1999 survey of 42744 physicians in Brazil (done by a pharmaceutical company in Brazil). The study revealed that 52% used the Internet -a level of diffusion equivalent to the general US population -however, when 23603 physicians users were asked from where they predominantly accessed the Internet, 85% indicated their home, 10% the office, and only 2% and 3% indicated the site as the university or the hospital respectively. In comparison, US physicians have the following Web access profile: 40% at the workplace, 56% at the office, 87% at home, and only 7% were not connected.
On a positive note, reform of the telecommunications sector is bringing significant improvement in services and drop in tariffs as a result of greater competition and expanding markets. With the recent rapid trade liberalization and modernization of the telecommunications sector in Latin America and the Caribbean, the telecommunications infrastructure is improving. One-fourth of the 89 major public telephone operators that were privatized throughout the world by the end of 1999 were in Latin America and the Caribbean [17].

Impact in the Society and in Health Practice
Many market segments are becoming increasingly information-technology dependent as part of globalization [7,[22][23][24]. Areas of concern in the introduction of electronic marketplaces, particularly in developing countries, are related to the difficulties in regulating offshore business, the dominance of the Internet global-communications infrastructure by a few countries, and growing concentration of power and knowledge in few corporations.
As is usually the case with innovation, the agents that first move into the market quickly attain a dominant position, block the entry of new competitors, and capture a large part of potential proceeds. It has been stated that the success of developed countries, particularly the United States, in taking advantage of ICT partly reflects its flexible and competitive markets. Possibly, smaller benefits can be expected in more-regulated economies or in the case of implementation environments characterized by rigid labor and trade rules and by inefficient commodity markets and capital exchanges [25].
Cross-border challenges are particularly pressing due to the growing number of national, international, and nongovernmental actors involved in transnational and global concerns. Market capture by strong, organized, and well-funded health-provider organizations, some of international nature, is happening at a fast pace in Latin America and regulatory methods have been advocated to safeguard local competition. Intangible health "e-Solutions" products and services offered by foreign providers -as is the case with investment, insurance, knowledge dissemination, and health care applications -present great challenges to developing and poorly-developed countries and may result in flight of capital, tax evasion, employment reduction, capture of the health market, and "cultural colonization." In the area of information technology, the emphasis of intellectual rights has changed from the protection of the author/inventor to that of the investor. Implications for developing countries [26], welfare effects, foreign direct investments, transfer of technology, and impact on domestic markets are difficult to foresee particularly in relation to foreign direct investments and technology transfer. Even countries with significant technology exports show negative net balance of royalties and license fees ( Table 6).
Intellectual property rights have been a major area of concern and conflict. The promotion of innovation is essential to any development strategy, particularly to increase international competitiveness of national enterprises. However, limiting acquisition of innovative technology only to those that are captured by the patent system ("inventions") makes a society permanently dependent on external sources. The universalization of standards for protection of intellectual-property rights has been enabled by the World Trade Organization General Agreement on Trade and Services (GATS) adopted in 1995 at the Uruguay Round of negotiations, reinforcing protection in 3 key information technology areas: computer programs, databases, and design of the layout of integrated circuits. Stakeholders concerned with such issues include nation-states; multinational business organizations; subregional trade blocks, and integration groups (formal trade blocks such as NAFTA [North American Free Trade Agreement], Mercosur, and the European Union Common Market, as well as other regional integration initiatives such as Andean Countries and CARICOM [Caribbean Common Market]). In addition, country initiatives, professional and nongovernmental groups, and international organizations provide an operational and legal framework for tackling these issues. Some of these entities are being increasingly overwhelmed by market access challenges not envisioned before the diffusion of ICT.

Skilled and Committed Human Resources are Essential
People are central in the value-added creation of eHealth products and services and an organization's human resource is the key to success [3]. Systems professionals, technology products and services providers, and project teams must also have superior skill levels and experience in the particularities of the area being automated. Regarding the number of technicians, scientists, and portion of the GNP devoted to research and development, the region is marginally better than other underdeveloped areas (Table 7). Employees' skills are the most-expensive and least-elastic resource and an obstacle to technological development in the region. The most-successful efforts to incorporate information and communication technologies in Latin America and the Caribbean have occurred in countries with strong and efficient government and academic institutions committed to invest in education, scientific and technological development, and public services, in tandem with business sectors (for instance, banking and retail commerce) ready and willing to automate their operations.

Public Health Authorities Frequently Have a Misguided Vision of ICT
Many public health organizations in Latin America and the Caribbean are not taking advantage of existing ICT opportunities and most existing information systems are inadequate to meeting the requirements of the new models of health care being deployed in the context of health-reform initiatives. Besides the common perception among physicians that health-information systems are mostly a source for scientific and technical information, often public health authorities have a view of clinical-administrative information systems that is obsolete and frozen in a "statistical-epidemiological" archetype, designed for the collection of numerical data representing only counts of events and mostly generating only highly-aggregated statistical data and time series related to mortality, morbidity, and to service utilization and coverage. Those information systems have very little practical interest to direct-care professionals and unit managers and are far behind in providing the logistical and longitudinal individual client-based data required to operate and manage the sort of health care models being deployed in many countries.
Worse still, most public health authorities are totally oblivious to the broad variety of possibilities offered by modern information and communication technologies to manage client-based data, support operations, and mine large databases. Indeed, the health sector has not applied the range of options provided by information and telecommunication technologies as effectively as have other social sectors, and health has been conspicuously underrepresented in national technology-development policies and plans. Such concerns have also been raised by traditional national statistics organizations [27].
As a counterpoint to the passiveness of the public sector, private providers and health groups recognized that a "different" type of information system and data elements are required to run their organizations and survive in a competitive environment driven by increasing consumer demands and expectations and to deliver personalized evidence-based services. Besides using ICT resources to boost productive specialization (such as allowing the efficient use of diagnostic services and consultations, maintenance of integrated records, reduction in the number of specialists, and attainment of economies of scale by linking to national and international markets) there are many new areas of application that are rapidly gaining ground and reducing care costs while improving the continuity and quality of care [28][29][30]. The lack of involvement of Latin American and Caribbean public-sector stakeholders in the use of ICT is worrisome. At a time when, in many countries, the ailing, bureaucratic, and inefficient public sector is struggling against poorly-regulated privatization of social services, there is a clear danger in that their inaction in adopting ICT solutions may hasten the further reduction and even the demise of public health services incapable of competing with an information-technology enabled private sector.

Fast Changing Environments, Untried Business Models, and Excessive Expectations in New Technologies and Processes
Resources, products, and markets that are highly specialized, closed, and regulated are being swiftly opened to new players in a marketplace that is still mostly unregulated and, at the same time, when novel and untried health reform models are being introduced. These circumstances carry with them a very high unpredictability of outcomes. Although businesses and public organizations are adapting with varying speed to new processes and models, the organizational "culture," and the nature and frequency of those business-environment changes may create friction, desirable and undesirable impacts, and personal behaviors that may impede the sequence of the expected project results. Broad objectives are difficult to achieve, the best strategy is to identify the most-repetitive tasks associated with significant costs -eg, the automation of claims and reimbursement procedures -and then proceed area by area. The results of the experience with e-Commerce and e-Business over the last 2 years clearly show that the emergence, adaptation, and real-world deployment of new technologies is a complex issue teeming with uncertainties. For a number of reasons, chiefly related to the technology employed, even in the most-industrialized and computer-literate societies e-Commerce has not developed smoothly.
Unfounded vendor-driven expectations of how the Internet will revolutionize health care have too often overshot their target [31]. Overestimation of results and consequent unfounded expectations is a common pitfall. A common error has been to regard technology as the solution for logistical, administrative, and knowledge-management problems of health care. While, at one extreme one finds technological pessimism and distrust of computer-based solutions, and even some hardcore Luddites driven by the digital-divide concerns, at the other extreme there are truly-excessive and extravagant expectations. The allurement of zero inventory, the oversimplified understanding of business processes, and far-fetched business models with thin or absent margins of return and costly customer acquisition and maintenance strategies have led many e-Companies to bankruptcy. The proliferation of e-Commerce sites of every conceivable nature, clearly not economically viable, resulted in the risk-capital investment bubble that ushered the catastrophic global technology-market collapse of the past year, which also slowed down the deployment of ICT applications in general in the region.
The lesson to be learned for eHealth is that technology is a tool, which can be justified economically only if organizations deploy it in a real-practice environment and closely track how managers and direct-care professionals are using it. This requires the stepwise development and implementation of processes and metrics to monitor productivity and impact [3,31,32].

Cost Concerns
Web enabling business and government operations is expensive. The United States can be used as a case example: Internet-based marketplaces can lower operational costs and improve efficiencies but deployment expenses will ordinarily cost a typical business US $5.4 million to US $23 million over 5 years.
Required procedures involve changing procurement processes, integrating online and internal systems, buying applications, and paying transactions fees and intermediaries. In general, such costs have the following distribution: 32% for internal preparation; 26% for initial contracts and fees; 20% for ongoing internal management; and 22% for ongoing fees and external services [33]. It is difficult for health executives, particularly in the public sector, to justify such levels of investment. There is no data for health ICT expenditures in the region but estimations for all sectors are summarized in Table 8.

Standardization is a Prerequisite
As providers and insurers soon realized, the simple automation of current processes and services and putting them in a Web-enabled environment is not feasible [3]. A great amount of work has been done in the creation and promotion of data-related standards [12] and, despite the lack of standards in some areas, fortunately there are solutions that allow different organizations and systems to communicate through standardized open-access Internet software languages. Process and data standards for the health care industry involving all constituents -employers, consumers, providers, payers, and regulatorspromoted by accrediting organizations have facilitated the adoption of common procedures and routines. A certain amount of standardization also has been driven by regulatory action. In the United States, the introduction of the Health Insurance Portability and Accountability Act (HIPAA) regulations forced a reluctant health industry to adopt uniform formats for health-data exchanges and uniform code sets to identify internal and external health services activities and to be HIPAA-compliant became a requirement of all applications. However, even in developed countries the lack of national standards for unique person identification has slowed implementation of patient-based information systems. An extensive review and reference source on health care data standards was published by the Pan American Health Organization [12].

Security and Privacy are Major Concerns
Data security and privacy of personal health data are universal concerns and a high-priority issue in many countries. There is a growing concern regarding the protection of health records against intrusion, unauthorized use, data corruption, intentional or unintentional damage, theft, and fraud. Health data transmitted over national and international networks offer unprecedented opportunities for better patient care and community health interventions by facilitating data exchange among professionals but pose new challenges to confidentiality. The promise of the Internet to improve care by timely access to the right information can only be realized through secure connections shared across all platforms.
Given the sensitive nature of health care information, and the high degree of dependence of health professionals on trustworthy records, the issues of reliability (data residing in the electronic health record is accurate and remains accurate), security (owner and users of the electronic health record can control data transmission and storage), and privacy (subject of data can control its use and dissemination) are of particular significance and must be clearly and effectively addressed by health and health-related organizations and professionals. Reliability, security, and privacy are accomplished by the implementation of a number of preventive and protective policies, tools, and actions that address the areas of physical protection, data integrity, access to information resources, and protection against unauthorized disclosure of information [34,35]. A comprehensive review and reference source on personal data protection regulation was published by the Pan American Health Organization [34].

Quality of Publicly Available Information
This is probably one of the most serious issues in the area of Internet-based interactive health communications. The Internet offers unprecedented power to provide all users of health care information -patients, professionals, families, caregivers, educators, researchers, insurers, regulators, and policymakers -with data of unprecedented timeliness, accuracy, depth, and diversity. Yet it is equally clear that the very qualities that make the Internet such a rich marketplace of ideas -its decentralized structure, its global reach, its leveling of access to the tools of publication, its immediacy of response, and its ability to facilitate free-ranging interchange -also make the Web a channel for potential misinformation, concealed bias, covert self-dealing, and evasion of legitimate regulation. It is very difficult to ascertain and provide recommendations about the credibility, motives, sponsorship, and eventual conflicts of interest in the more than 50000 health Web sites in existence. Many websites are profit driven, others promote unproven and even dangerous forms of treatment or products, while others may be well intentioned, but contain misleading or false information [30,[36][37][38][39][40][41][42][43][44][45][46][47].

Recommendations on Policy and Organizational Issues for the Deployment of eHealth in Latin America and the Caribbean
The Presidential Declaration of the 1998 Summit of the Americas and a number of international meetings held since then (Florianópolis Declaration by the representatives of Latin American and Caribbean countries; Brasília Communiqué of the Presidents of South America; Rio de Janeiro Declaration of the Intergovernmental Meeting on ICT for Development; Declaration of the Rio Group; the Declaration of Santiago of the Rio Group; and the European Union Minister's Meeting) made clear that all countries of the Americas have a common stake in improving access to and delivery of health care through communications and information technology. The recommendations presented here follow the spirit of the principles agreed by the governments of the region in the above-mentioned high-level meetings.

Developing a National Vision, Mission, and Plan of Action for the Public and Private Sectors
The immediate objective is to promote the deployment of core eHealth applications and support functions by incorporating an advanced informatics component into health programs and projects, supported by a combination of funding programs, incentive grant programs, and prototype development funding programs. Elements for the development and implementation of a comprehensive national technology and policy vision, mission, and action plans for eHealth must address the following issues: • Develop a telecommunications infrastructure that is comprehensive, reliable, ubiquitous, and compatible across applications -such an infrastructure must provide affordable bandwidth that is sufficient to serve the wide variety of users' specific needs. Its development will be dependent upon the continued deregulation of the telecommunications industry and will involve the leveraged use of many ICT technologies that have been spawned by and for other industries.
• Provide technological interfaces that facilitate effective use of the infrastructure and its component systems -these interfaces may involve systems capable of rendering information from multiple modalities, in conjunction with a variety of applications as aids to health services operational support and decision making. They will require modularity and connectivity compliant with standardized interface protocols.
• Implement legal and regulatory infrastructure that will facilitate medical communication -at the professional level, such issues as interstate/province licensure and credentialing of service providers must be addressed, the telecommunications and information industries must develop rational and affordable tariff structures, and legislation must be passed to ensure the protection of personal health information.
• Implement rational and technologically-neutral policies for public and private payers -coverage and payment policies should be established to address the entire range of eHealth applications and technologies. Means should be developed for assessing the appropriateness of health services provided via telemedicine applications.
Outcome-based quality-improvement programs will be of great importance in assuring quality and cost-effective medical care.
• Make available appropriate content to consumers, patients, and service providers with the objective of enhancing health care outcomes -the process for conveying quality evidence-based information must permit the user to follow the links between data, inferences, and conclusions. Authentication, access control, confidentiality, integrity, and attribution are key requirements for health-related advice and decision making.
• A retrospective of experiences shows that continuity and sustainability of information-systems projects continue to be a major problem in the region. Externally-funded projects frequently collapse upon funding termination and this demonstrates that all projects need justification in terms of cost-benefit and long-term financial sustainability besides organizational capacity to develop and implement information systems. This further indicates that spreading the financial risk across several stakeholders may be appropriate as cost sharing increases overall awareness, utilization, and long-term potential for success.
• Standards development and implementation must be carried out with the public and private sectors to achieve consensus on a set of information principles for the collection, transfer, storage, and use of health data over national and global information infrastructures. Standards will be defined by a consortium of users; researchers; government, technical and scientific bodies; and the industry at 3 distinct levels: first, in terms of standardization of data and information; second, in terms of the computational facilities required to manipulate and store the information; and third, in terms of telecommunications facilities, employed to transfer information among dispersed sites.
Six areas are envisioned for government involvement in eHealth development and deployment: (a) promotion of education, training, and national planning capacity in information systems and technology; (b) convening groups for the implementation of standards; (c) providing funding for research and development; (d) ensuring the equitable distribution of resources, particularly to places and people considered by private enterprise to provide low opportunities for profit; (e) protecting rights of privacy, intellectual property, and security; and (f) overcoming the jurisdictional barriers to cooperation, particularly when there are conflicting regulations.
The attainment of this mandate involves participation of a large number of stakeholders, but the coordinating effort will necessarily concentrate on the public sector.

Bridging the Digital Divide
Only a more-active role of government and public-private partnerships in supporting appropriate technology transfer and adaptation through indigenous research and development and the implementation of specific policies to protect local development will create an environment conducive to a reduction of the present ICT development divide. Increasing the general population's capacity to take advantage of information and communication technologies requires heavy investment in general education and training in computer skills. A serious problem for Latin America and the non-English speaking Caribbean is that most of the Internet is directed to native speakers of English and most sites and exchanges are carried out in English. Even physicians, who can generally be expected to have a working knowledge of English, may have problems with such sites. This means that investment is required to develop applications, user interfaces, and contents in national languages.
Developing countries of Latin America and the Caribbean may take advantage of the accumulated knowledge and mistakes and may leapfrog developmental stages; however, this is not expected to be readily achieved due to the barriers posed by the general institutional underdevelopment, low income, illiteracy, and financial constraints that afflict many countries. It is improbable that the bridging of the health-sector development divide will be accomplished easily. In industrialized countries, it took several decades and countless institutional and organizational transformations for the consolidation of economic, institutional, and technological changes and the crystallization of long-term structural patterns necessary for information and communication technologies to spread to vast sectors of the society.

Developing Organizational and Human Resources: Awareness, Skills, and Leadership
The current health-sector organizational structure and national regulatory framework in Latin America and the Caribbean are not conducive to problem-oriented, interdisciplinary, rapid-response collaborative technical work, and the implementation of political, regulatory, and managerial tasks required to address multifaceted complex technological problems. Organizational and human-resources development through awareness programs, education of health staff, continuous training, and career opportunities must be institutionalized from the inception of the developmental effort. Transference of technical expertise and the appropriation of knowledge by health personnel are necessary for the full participation of end-users in the development process and the best insurance for successful implementations. Success in the deployment of institutional eHealth applications depends on the existence of staff with the right mix of skills in all functions and levels. Recommended strategies include: • A structured human-resource development program defined with the goal of increasing awareness of eHealth opportunities and training health professionals to assume a leadership role and actively participate in all aspects of systems design and implementation.

•
The training strategy will take into account issues associated with: the development, the organizational environment in which systems are expected to operate, and the specific circumstances of the local health system. The following guidelines for training should be implemented: identify target groups based on functions and training needs; develop training programs to meet identified target groups' needs; and establish a network of training focal points, taking into account the specific organization and circumstances of national characteristics and local health-unit requirements and undertakings.
• Target groups to be considered are: those who originate, collect and supply data; operational decision makers (direct health care professionals and administrators); managers, planners, and policy makers; information systems managers; information technology and computing specialists; data analysts; and statisticians and researchers.
• Each country will develop its own strategy for initial and continuing training in health-information systems, considering the overall development of health information systems and its particular health care, educational, research, and market environment.

Financing and Public-Private Partnerships
Given that the worldwide market for information technology, products, and services in 2001 was valued at US $1086 billion and expected to grow 32% to US $1436 billion by 2004 [48], developing countries need to find ways to share this growing trend. Domestic and foreign, public and private investment sources will be involved, ranging from revenue-sharing initiatives and joint ventures to direct investment, transfer schemes, development funds established by a special tax on telecommunications, major private financial institutions, loans from international funding agencies and development banks, and incentive grants. Joint investment and development involving users, governments, academic and financing institutions and agencies, technical cooperation agencies, and industry interests are seen as necessary. Partnerships with the informatics industry are absolutely fundamental and, in the case of general informatics tools, the industry practically drives the solutions. A concerted effort is needed to secure a clearly-defined and specified partnership with the informatics industry at the global and national levels aimed at application development at acceptable cost. Investments must be attracted to the telecommunications industry by improving investment conditions, lowering duties on telecommunications equipment, and posing no restriction on network design except for technical reasons to allow for new providers.
Development of Health Informatics in the region must be conducted in the context of a framework linking public, private, and social efforts to speed the development of priority ICT solutions. Technical knowledge, experience, and financial investments needed to establish large and complex information system projects require tapping into resources and expertise that no single organization retains. Public and private institutions, academic organizations, the industry, and financing agents must find ways to pool their assets through project partnerships and to add social value to applications of informatics by providing new employment opportunities, socioeconomic development, and educational opportunities, by promoting health, and by supporting cost-effective health services.

Fostering International Cooperation
In the international setting, cooperation between developed and less-developed countries is essential but special care must be taken to avoid interventionist behavior that ignores users' real needs, fails to understand host capacities, demands action without allowing sufficient time for conceptual assimilation, neglects cultural constraints, and ignores hosts' knowledge basis. As in many other areas of international cooperation the danger is to have too much too soon or too little too late. A possible framework for collaborative work should include support to international health issues, health care reform implementation, application development, education, and economic and technological cooperation.
By demonstrating that social projects, especially health care and education, can be advanced through improved information infrastructure, international technical cooperation and multilateral agencies must collaborate with national and international authorities and experts to demand that funding institutions finance projects in such areas. Consistent with these objectives, governments must demand that international and multilateral agencies must promote and support technical cooperation activities in the development of eHealth, primarily involving knowledge transfer, technical support, facilitation of the exchange of experiences between countries, and fostering the use of appropriate technology and knowledge assets. Priority areas for technical cooperation include: priority assessment, technology evaluation and selection criteria, implementation issues, emerging technologies linking patients and providers, access to knowledge databases, consumer informatics, and the utilization of Internet and Internet-enabled technologies.

Creating Incentives Through Regulation
Many Latin American and Caribbean countries are committed to reform their telecommunications systems. They recognize that progress in the telecommunications sector is essential to establishing health informatics and to ensuring the global competitiveness of their economies, and attention has been on liberalizing the markets. Recommendations include actions in the following areas: market access issues (interconnection-regulation framework, clear and transparent regulation governing competition, and allocation of spectrum harmonization [ie, assignment of the same frequency bands everywhere]); standards (interoperability standards and the streamlining and liberalization of the conformity-assessment process for equipment certification); regulation (elimination of rules of origin and treating products from different countries equally when standards are the same, the elimination of subsidies, antidumping practices, and abolishment of countervailing duties); promoting competition (establish a regulatory framework that balances national needs in the context of creating a competitive national telecommunications system, weigh cost of delaying competition against the need for an effective transitional regime, and move towards full liberalization as quickly as appropriate); and protecting technology and intellectual-property rights.
Experts agree that to be effective and efficient the health care industry must operate in a digital environment, encompassing connectivity, commerce, and community/content sites. But no one can pinpoint exactly when everything will converge. Nudging the health sector toward compliance is a valid and effective approach. The European and Canadian health care systems have used this strategy and HIPAA, the US federal Health Insurance Portability & Accountability Act, is a prime example of how the industry can be coached into complying with a variety of guidelines related to standardization, security, and privacy. In effect, HIPAA is forcing an eHealth solution on the US health care industry.

Conclusions
Health care organizations in Latin America and the Caribbean, particularly in the public sector, are not yet prepared to adopt ICT effectively. The goal of a national health ICT vision and strategic plan of action is to establish a coherent national arrangement directed to facilitating projects and infrastructure development, maximizing the benefits for invested financial resources, and enabling people to accept and function more effectively in an informatized, evidence-based, and competitive health practice environment.
Governments should focus on their role as sponsors of basic scientific and technological research, bridging the digital divide, fostering public-private partnerships, managing international cooperation efforts, and establishing the regulatory and incentive components. All stakeholders must work collaboratively to grapple with the many standardization and infrastructure-development issues and the transnational and global eHealth aspects that must be addressed in a comprehensive manner. International aspects of eHealth services form a critical and urgent area still to be addressed by the World Trade Organization and regional trade blocks. Legislation proposals should be initiated to ensure that the technology does not abridge patients' rights to confidentiality or security of medical records, and that agreement on practice parameters be developed to include aspects related to informed consent, physician liability, nonphysician liability, reimbursement, practice standards, and physician-patient relationships.
The public sector, the industry, and partnerships have the responsibility for assuming an active leadership role in educating the medical community and in coordinating and encouraging the effective implementation of relevant applications. Health organizations must be provided with information about the opportunities as well as the risks of eHealth solutions. Technology-evaluation sources and results must be made available and health managers must be guided in the difficult process of specifying systems, procuring, acquiring, and contracting for ICT products and services. Knowledge repositories must be established in cooperation with the industry, centers for technology evaluation, academic research groups, and centers of excellence.

Introduction
One crucial way to provide online access to novel, valuable, and quality-assessed research reports is to foster open access to the peer-reviewed primary research literature. Examples of recent initiatives in this area are the Budapest Open Access Initiative (BOAI) [1] and the Free Online Scholarship movement [2].
To achieve open access to scholarly journal literature, the BOAI has recommended 2 complementary strategies [1]. The first is self-archiving, by researchers and scholars, of their refereed journal articles. The second is the fostering of open-access journals. Peter Suber, the editor of the Free Online Scholarship Newsletter [2], has argued that various objections (eg, that open access to scientific journal literature requires the sacrifice of peer review, revenue, copyright protection, or other strengths of traditional journals) are based on misunderstandings [3].

What criteria could be used to evaluate the success of the BOAI?
Several criteria that could be used to evaluate the success of the BOAI have been summarized [4]. For example, one indicator, oriented toward the first of its strategies (self-archiving), is the number of universities (or analogous organizations, such as research institutes or networks) that have created institutional e-print archives and have adopted policies that encourage faculty to use them. A comprehensive list of various kinds of e-print archives has been assembled by the Virtual Technical Reports Center of the University of Maryland Libraries [5].
Another indicator, oriented toward the second BOAI strategy (more open-access journals), is the number of journals in each research field that are committed to open access, coupled with measures of the reputation or impact of each journal. A list of open-access medical journals is available from the Free Medical Journals website (JMIR is included in this list) [6]. the free, full-text research literature [14]. These methods can be expected to yield novel approaches to assessments of the impact of individual research reports and to explorations of linkages among related research reports.
From the agency's perspective, opportunities to assess, over the longer term, the outcomes of research projects the agency has funded should have considerable appeal. But, to avoid becoming (in effect) a journal publisher, the agency might be well-advised to permit grantees to post research reports at its e-print archive only temporarily. For example, perhaps the lifetime of any particular preprint in the archive should end at either 5 years or when it has been published in an appropriate journal (whichever comes first). Ideally, the agency would also maintain a publicly-accessible database of the research projects and programs that it has funded. It would include, in this database, links to the abstract (and, if possible, to the openly-accessible full text) of any published reports that emanated from these individual projects or programs. And, whether or not the agency decides to become the equivalent of a journal publisher, it should establish e-print archives that are not only openly accessible, but also interoperable [15] (interoperability allows searches across free, full-text research-literature e-print archives; the results can be ranked according to many criteria, eg, by citation impact [14]).
Incentive models of various kinds may become examples of a "little thing that can make a big difference," of the kind described in Malcolm Gladwell's book, "The Tipping Point" [16]. A "tipping point" is "that magic moment when an idea, trend, or social behavior crosses a threshold, tips, and spreads like wildfire." Open access to the peer-reviewed primary biomedical and health research literature has not yet reached a tipping point. Perhaps, via an appropriate mix of strategies of the kind outlined above, it soon will.