Published on 10.11.05 in Vol 7, No 5 (2005)
What Is eHealth (5): A Research Agenda for eHealth Through Stakeholder Consultation and Policy Context Review
Background: In 2003, the National Health Service in England and Wales, despite its large investment in information and communication technology, had not set a national research agenda. The National Health Service has three main research and development programs: one is the Service Delivery and Organisation program, commissioned in 2003, and the others are two parallel “scoping exercises” to help set a research agenda. This paper reports on one of those projects. A parallel literature review was carried out by others and has been reported elsewhere.
Objective: The objective was to explore the concerns of stakeholders and to review relevant policy in order to produce recommendations and a conceptual map of eHealth research.
Methods: There were two parallel strands. For the stakeholder consultation, 37 professionals representing 12 “stakeholder” groups participated in focus groups or interviews. Discussion was prompted by eHealth “scenarios” and analyzed using thematic content analysis. Subsequently, 17 lay participants, in three focus groups, discussed and prioritized these themes. For the policy review, 26 policy makers were interviewed, and 95 policy documents were reviewed. Recommendations were subsequently reviewed in a conference workshop. Recommendations for research from both strands were combined into a conceptual map.
Results: Themes from stakeholder consultation and policy review were combined as 43 recommendations under six headings. Four of these headings (using, processing, sharing, and controlling information) describe the scope of eHealth research. The other two relate to how research should be carried out (ensuring best practice is first identified and disseminated) and to the values considered important by stakeholders (in particular, measuring improvement in health).
Conclusions: The scope of eHealth research (using, processing, sharing, controlling information) derived empirically from this study corresponds with “textbook” descriptions of informatics. Stakeholders would like eHealth research to include outcomes such as improved health or quality of life, but such research may be long term while changes in information technology are rapid. Longer-term research questions need to be concerned with human behavior and our use of information, rather than particular technologies. In some cases, “modelling” longer-term costs and benefits (in terms of health) may be desirable.
J Med Internet Res 2005;7(5):e54
In 2002, the National Health Service (NHS) in England and Wales planned to invest over £2 billion in information and communication technology (ICT) . This includes initiatives such as electronic patient records, electronic prescribing, the NHS Direct Telephone and Internet Service, and the National Electronic Library for Health [ , ]. Researchers from multiple disciplines in the UK and elsewhere had been investigating health informatics, but the NHS, despite its large investment in ICT, had not set a national research agenda for ICT.
A one-day conference in 2002 on Health Informatics Research and Development, sponsored by the research councils, Department of Health, and Department of Trade and Industry, concluded that the “lack of national strategy, capacity and career paths in health informatics have been weaknesses and remain threats to realising the informatics potential of the National Health Service…. [L]arge investment in the National Health Service and e-Science is unlikely to achieve its objectives without radical improvement in support for academic health informatics…. [This emphasizes the] importance of...clarifying the academic agenda for health informatics” . The short- and medium-term challenges were seen as the following: (1) establishing the foundations of a knowledge infrastructure, (2) innovations in the clinician computer interface, (3) workable privacy protection, (4) more creation of knowledge from routinely collected data, and (5) finding the metrics of success for health informatics.
The NHS has three main national NHS research and development programs: Health Technology Assessment, New and Emerging Applications of Technology, and Service Delivery and Organisation (SDO). The SDO was launched on March 30, 2000, to consolidate and develop the evidence base on the organization, management, and delivery of health care services . To respond to the needs of the “stakeholders” (service users, health professionals, and policy makers), the NHS, through its SDO research program, undertook an initial “listening exercise”[ ] to produce a document outlining its overall priorities for research. It has continued to use this approach to develop and commission research [ ]. It commissions a “scoping exercise” (normally a literature review and a stakeholder consultation) and then uses that in subsequent calls for proposals.
This study explored the concerns of professional and lay stakeholders regarding future developments of eHealth and reviewed relevant policy to produce recommendations for eHealth research. A parallel literature review was carried out by others and has been reported elsewhere [- ].
The study was reviewed and approved by the SouthWest Multi-Centre Research Ethics Committee and the University of Plymouth Faculty of Health and Social Work Ethics Committee. Data collection was carried out between November 2003 and June 2004. There were two parallel strands ():
- Stakeholder consultation: Focus groups and interviews with “professional” stakeholders generated themes that were subsequently prioritized by lay participants.
- Policy context review: Policy makers were interviewed and policy documents were reviewed in order to produce recommendations that were subsequently reviewed in a conference workshop.
The themes and research questions arising from the stakeholder consultation and policy context review were compared, and recommendations from policy context were adapted to take account of stakeholder concerns. Diagrams were developed to bring together stakeholder and policy maker views of the scope of eHealth research.
In all, 12 groups (30 in each group) of professional stakeholders were contacted via email. Potential participants were identified by Web searches and by “snowballing” from existing contacts, trying to get geographical coverage within England. Stakeholders were defined as the following:
- NHS eHealth innovators and implementers
- University researchers in health informatics
- NHS staff in primary care
- NHS staff in secondary care
- NHS primary care trust managerial staff
- NHS acute trust managerial staff
- Suppliers of ICT to the NHS
- Professional organizations and royal colleges
- Informatics trainers
- Governance and other regulators
- Charities and other information providers
- Other NHS managers
These 360 people were sent an email inviting them to take part in the study, with a consent form to return to the researcher via email or post if they agreed to take part in the study. We asked professional participants to rank themselves on a four-point scale (from “I am pretty sceptical that eHealth will have any benefits at all” through to “I am very positive that eHealth can help improve the NHS if used appropriately”) as a rough guide to attitudes toward eHealth ().
|Stakeholder Group (number of participants)||Location in England||Self-Rating on eHealth|
|Innovators and implementers (5)||South West||3|
|Academic researchers in eHealth (4)||Midlands||3|
|NHS staff in primary care (3)||South West||3|
|NHS staff in secondary care (4)||South East||3|
|Primary care managerial staff (1)||South West||3|
|Acute care managerial staff (1)||South East||3|
|Suppliers (3)||South East||3|
|Royal colleges (3)||South East||3|
|Informatics trainers (5)||South East||3|
|Governance (3)||South East||3|
|Charities and other providers (2)||Midlands||3|
|Other NHS managers (3)||South East||3|
* Self-ratings: (1) I am pretty sceptical that eHealth will have any benefits at all. (2) I think that there could be some possible benefits to eHealth methods but on balance think that it is unlikely that the benefits will outweigh the costs. (3) I think that there are definitely benefits to eHealth but that we need to choose and develop methods carefully. (4) I am very positive that eHealth can help improve the NHS if used appropriately.
A convenience sample of lay participants was recruited via snowballing from contacts in a local children’s nursery and from older friends of the research team. Potential participants were sent study information sheets and consent forms either via email or post. Two groups of older people and one group of parents took part in the study.
Scenarios depicting the current or future use of eHealth technologies were constructed to prompt discussion among the professional groups of the relevant themes regarding the use of eHealth technologies. Subject matter of the scenarios was developed from the content of news reports, informatics conference proceedings, and general Web searches. Both patient- and professional-centered scenarios were developed in order to achieve a balance of perspectives. The research team developed and discussed 32 scenarios: 15 were omitted, 7 were added, and a number were reworded to ensure neutrality in presentation.
In total, 24 scenarios (see) were allocated to the 12 professional groups using a balanced incomplete block design [ ]. Each group had four scenarios, and each scenario was used twice. A semistructured schedule based on the scenarios was constructed for use either as focus group or interview prompts. Some scenarios described patients being monitored by an implanted device that sent physiological information to hospital, a family doctor booking a hospital appointment during consultation in primary care, and a woman having an antenatal ultrasound in the community with expert diagnosis from abroad. provides four examples of scenarios used as prompts in telephone focus groups. The topics covered in the scenarios included patients’ use of the Internet to order prescriptions, arrange doctor’s appointments, or join patient discussion forums. Other topics were about patients accessing their own health record, assessing the quality of a website, using a digital interactive television for a program on multiple sclerosis, or using a public access touch screen health information point.
A parallel, two-stage process was used to review the policy context. First, policy makers were interviewed and policy documents were identified and reviewed. The contents of documents and interview notes were categorized under English policies on ICT specific to health, English health policies influencing eHealth, nonhealth policies influencing eHealth, and European Union policies influencing eHealth. The reviewers focused on seven specific topics: (1) birth to death records, (2) country-wide access to quality health advice, (3) application of ICT to pharmacy, (4) telemedicine, (5) reduction of adverse incidents, (6) confidentiality, and (7) health data cards. In a second stage, 28 recommendations from the policy makers and documents were reviewed in a health informatics conference workshop by 60 participants using discussion and an interactive voting system. Participants scored each recommendation for relevance to the needs of the NHS (using a nine-point scale from “not relevant” to “highly relevant”). Recommendations with “middle scores” were discussed with the audience in more detail to obtain their views and decide if the recommendation needed to be worded more clearly.
Synthesis and Conceptual Mapping
Two members of the research team (RJ and LC) independently ranked the correspondence between the stakeholder concerns and the policy context recommendations on a scale of 1 (no correspondence) to 3 (strong correspondence). Analysis showed there to be strong agreement between the researchers. Stakeholder concerns, particularly of “technology meeting needs and improving health and quality of life,” were not consistently addressed by the policy context recommendations. Recommendations were adapted to take stakeholder concerns into account and were regrouped from “source-oriented” to “research-oriented” groupings. The new policy context recommendations were agreed upon between team members by telephone conference discussion. The two lists, one from stakeholder consultation and the other from the policy context review, were then reviewed again and similar areas of research were grouped. Reference was also made to the work of the Scottish Consumer Health Informatics Network . We concluded that the scope of eHealth research could be described by a simple block diagram with four elements with linked areas of “best practice.” The recommendations were regrouped according to this “conceptual map.”
Subjects and Sources
In total, 37 (10%) professional stakeholders consented to take part in the study and were consulted either via telephone focus group (25), telephone interview (6), videoconference (4), or in-person interview (2). There were 17 lay people (12 older people and 5 parents of young children) that took part in in-person focus groups. We interviewed 26 policy makers and identified and reviewed 95 policy documents.
Validity of Methods
One of the limitations of stakeholder consultation is that participants have to be sufficiently interested in the topic to take part. None of the 37 professional participants were “sceptical that eHealth will have any benefits at all,” and none thought “there could be some possible benefits to eHealth methods but on balance think that it is unlikely that the benefits will outweigh the costs.” On the other hand, 27 thought “that there are definitely benefits to eHealth but that we need to choose and develop methods carefully,” and 10 were “very positive that eHealth can help improve the NHS if used appropriately.”
Scope of eHealth
The research questions identified by stakeholders and policy review fell into six groups. Four of these (using, controlling, processing, sharing information) were used to describe the “scope” of eHealth. The other two groups of research questions fall under principles of research and development and stakeholder hierarchy ().
Principles of Research and Development
Both stakeholders and policy makers referred to many examples where, before innovative approaches are introduced, best practice procedures and barriers to implementation should be identified, and where professional and public stakeholders should be involved in research and development. A number of areas were suggested ().
Scope of New Research in eHealth
Information is used in decision support, in the organization of services, for reassurance of professionals and patients, and in information-based therapies. Four research aims () from the policy context review concerned the use of information in decision support.
Both stakeholders and the policy context review identified a large number of research questions related to sharing information (). These included how information should be shared across sites (eg, between hospital and home), across sectors (eg, between social services and NHS), and between different professional (and patient) groups (eg, between doctors, nurses, dentists, patients).
This group of concerns was ranked second most important by stakeholders. Ten research aims () incorporated issues of control, accessibility, reliability, confidentiality, security, ownership, and regulation.
This covers a range of issues, including how best to present information (eg, should it be tailored for different users) and where it might be presented (eg, should it be sent to the user [push], or should it wait until the user seeks it [pull]). It also includes the coordinated integration of information derived from a variety of sources, as demonstrated in the electronic ordering and home delivery of medicines ().
Hierarchy of Stakeholder Concerns
The overriding concern of stakeholders was that spending money on eHealth should be worthwhile and should lead to improved health and quality of life. Particular research aims suggested by the data included “to review the costs and benefits of a range of recent eHealth applications, including the modelling of new forms of care made possible by ICT support,” and “to present those examples of eHealth applications, shown to have a demonstrable effect on improved health and quality of life, to professional and public stakeholders to obtain their views as to the nature of the most appropriate investment in eHealth.” In addition, the stakeholders identified themes concerned with controlling information (responsibility; reliability; regulation; accessibility; confidentiality, security, and ownership) as being particularly important, and so placed them on the second level of a hierarchy of concerns.
In the context of the British health service, which is mostly free at the point of delivery, the overriding concern of stakeholders was that spending money on eHealth should be worthwhile and should lead to improved health and quality of life. At first this appears an unremarkable finding. However, although such an aim is part of the political rhetoric and may be an unstated assumption of policy documents in the United Kingdom, it is not often explicitly addressed in service development and use of information and communication technology. It is significant, for example, that the NHS recently funded a program originally called the National Programme for Information Technology, rather than a program for eHealth. (Subsequently, the program has been renamed Connecting for Health).
On the other hand, we know that doing research that can show a difference in health as a result of an eHealth intervention is difficult, partly because partial implementation of an e-booking system or a hospital information system is difficult. Gold standard randomized trials whose results can be generalized for widespread implementation are very difficult. In addition, to see changes in health or to measure cost benefit is slow and expensive, made more so now in the United Kingdom by the time needed for increased levels of ethical and research control and approval [, ]. Thirty years ago, Blum noted that half the papers about computer applications concerned systems that were no longer operational [ ]. We all know that ICT changes become ever more rapid. Research, therefore, has to be more about human behavior and how we use information and less about specific organizational or technological environments. It is essential to recognize the difficulties of addressing stakeholders’ needs by measuring change in health outcomes.
In some cases, modelling the longer-term costs and benefits (in terms of health) may be desirable. As systems continue to evolve, the health benefits may be seen not in the immediate change, but in a future evolution made possible by the initial change [- ].
Many of the recommendations derived from both the stakeholder consultation and policy review confirmed the need to identify best practice and the barriers to implementation of that best practice. There are many examples of medical informatics research with demonstrable benefit which decades later still wait to be implemented more widely. For example, computers have been used successfully for patient interviews for nearly 30 years. Slack et al first reported on a computer-based medical history system in the New England Journal of Medicine in 1966 . Yet, despite numerous research reports [ ], the method has not been routinely adopted.
Rogers’ description of the diffusion of innovations  is well known, but stakeholders and policy makers want to see this process accelerated. The recommendations included, for example, identifying and exploring examples of effective informatics training for health professionals to see how these can be disseminated, or investigating working practices in other sectors (eg, e-business) to identify best practice and barriers to similar uses of ICT in the health sector.
In discussion, professional stakeholders often reverted to their role as patient or consumer rather than, for example, speaking as a supplier of ICT or from the point of view of primary care. Thus, although we sampled from all segments of the stakeholder population, there were no obvious differences between the different types of stakeholder. (Our sample was small, however, making our power to detect differences limited.)
The four categories which emerged as the grouped themes from the data are similar to classifications and descriptions found in textbooks of health informatics (eg, ). Blum, in a personal review of Medical Informatics in the United States, 1950-1975, presents a historical table of the Scope of Medical Computing from 1950 to 1980 using the three headings Data, Information, and Knowledge Applications [ ]. He said, “Within a category, research begins only after the supporting technology is mature enough to support it beyond the conceptual level.” His table showed that he viewed data applications in the 1980s as refined, information applications as mature, and knowledge applications as prototype.
Four limitations of our study are the following: (1) We were not able to recruit people who were very sceptical about the potential of eHealth. However, we think that a more sceptical sample would be likely to have expressed similar concerns about improved health and value for money. (2) We did not have equal representation for the 12 predefined groups. However, we achieved coverage (although, in some cases, only one member; see) for all groups. Furthermore, as most participants often “reverted” to their role as patient in the discussion, our original idea that different professional groups might have certain biases and try to put forward ideas to their advantage seemed wrong (although numbers were small to detect any differences between groups). (3) Our lay sample was recruited only in the South East and the South West of England and did not include any people from ethnic minorities. We have no knowledge or hypotheses about how views may differ in groups not represented. (4) An eHealth agenda derived from policy review and stakeholder consultation in a country with a state-run health service may not transfer well to countries with private or insurance-based systems.
The purpose of our study was to produce a conceptual map and research agenda for eHealth based on stakeholder views and policy review. In a parallel SDO project, Pagliari et al [- ] developed recommendations based on a review of existing academic and wider evidence sources, indicating the scope of the eHealth concept, the effectiveness of eHealth innovations, issues for implementation, and future directions for eHealth. Their research evidence is grouped by four broad technological categories:
- Decision support tools for patients and clinicians
- Networked digital technologies (Internet) used by patients (eg, for information, self-management, or peer support) and professionals (eg, for interprofessional communication, education, or communication with patients)
- Computerized patient records, including issues relating to patient access and confidentiality and influences on health care delivery
- Telemedicine and telecare
Their results are also interpreted with respect to the following broad content areas:
- Specific research needs (evidence of effectiveness in specific areas)
- Generic research needs (eg, methodological challenges to eHealth research, factors affecting implementation, effects on behavior and relationships, educational interventions, health inequalities, alternative delivery media, risks to the health service and society, role in self-care, and consumer empowerment)
- Challenges for implementation (demonstrating impact, high-level support, strong project management, stakeholder engagement, the digital divide, ensuring credibility and quality, ethical, security and privacy issues, standards)
- Emerging trends and future directions (eg. personalized and tailored systems, new technological advances for the information management and care facilitation, and delivery modes)
While differences in emphasis were expected due to the methodology of each study, the clear parallels between the results offer support for our recommendations. Our conceptual map, which has come from stakeholder discussion and policy review, also helps to put both our own and other detailed recommendations into a framework concerned with information and how we use it.
We thank all those who participated in this study. The study was funded by the NHS SDO Programme.
Conflicts of Interest
Multimedia Appendix 1
List of 24 scenarios. PDF file, 92 KB
- ; Department of Health. Delivering 21st Century IT Support for the NHS: National Strategic Programme. London, UK: Department of Health Information Policy Unit; 2002.
- ; Department of Health. The NHS Plan. A Plan for Investment. A Plan for Reform. London, UK: The Stationery Office; 2000.
- ; Medical Research Council. Proceedings of the Workshop on Health Informatics Research and Development; July 18, 2002. London, UK: Royal College of Obstetricians and Gynaecologists. URL: http://www.mrc.ac.uk/pdf-health_informatics_r_and_d_workshop.pdf [accessed 2005 Jul 23]
- ; NHS Service Delivery and Organisation (SDO) Programme. Home page. URL: http://www.sdo.lshtm.ac.uk/index.html [accessed 2005 Jul 23]
- Fulop N, Allen P. Service Delivery and Organisation National Listening Exercise: Report of the Findings. London, UK: National Health Service Executive; 2000.
- Lomas J, Fulop N, Gagnon D, Allen P. On being a good listener: setting priorities for applied health services research. Milbank Q 2003;81(3):363-388. [Medline]
- Pagliari C, et al. EH1 E-Health Scoping Exercise. Review of the Traditional Research Literature. URL: http://www.sdo.lshtm.ac.uk/ehealth.htm [accessed 2005 Jul 23]
- Pagliari C, et al. EH1 E-Health Scoping Exercise. Review of the Non-Traditional Research Literature. URL: http://www.sdo.lshtm.ac.uk/ehealth.htm [accessed 2005 Jul 23]
- Pagliari C, Sloan D, Gregor P, Sullivan F, Detmer D, Kahan JP, et al. What is eHealth (4): a scoping exercise to map the field. J Med Internet Res 2005 Mar 31;7(1):e9 [FREE Full text] [Medline] [CrossRef]
- Armitage P, Berry G. Statistical Methods in Medical Research. Oxford: Blackwell Scientific Publications; 1987.
- Marsden J, Jones R. Establishing and maintaining the Consumer Health Informatics Network for Scotland (CHINS). Health Care Computing 2004:169-177. URL: http://www.health-informatics.org/hc2004/P27_Marsden.pdf [accessed 2005 Sep 26]
- Elwyn G, Seagrove A, Thorne K, Cheung WY. Ethics and research governance in a multicentre study: add 150 days to your study protocol. BMJ 2005 Apr 9;330(7495):847. [Medline] [CrossRef]
- Hearnshaw H. Comparison of requirements of research ethics committees in 11 European countries for a non-invasive interventional study. BMJ 2004 Jan 17;328(7432):140-141 [FREE Full text] [Medline] [PMC] [CrossRef]
- Blum BI. Medical Informatics in the United States, 1950-1975. In: Blum BI, Duncan K, editors. A History of Medical Informatics. Proceedings of the ACM Conference on the History of Medical Informatics; Bethesda, Maryland; November 5-6 1987. New York: ACM Press; 1990:xvii-xxxx.
- Friedman CP, Wyatt JC. Evaluation Methods in Medical Informatics. London, UK: Springer-Verlag; 1997.
- Mumford E. A socio-technical approach to systems design. Requir Eng 2000;5(2):125-133.
- Jones RB, Hedley AJ. Evaluation of a diabetes register and information system. In: Richards B, editor. Current Perspectives in Health Computing. Weybridge, UK: BJHC Books; 1986:80-87.
- Slack WV, Hicks GP, Reed CE, Van Cura LJ. A computer-based medical-history system. N Engl J Med 1966 Jan 27;274(4):194-198. [Medline]
- Jones R, Knill-Jones RP. Electronic patient record project: direct patient input to the record. Report for the Strategy Division of the NHS Information Management Group. Glasgow: University of Glasgow; 1994.
- Rogers E. Diffusion of Innovation. 4th ed. New York: The Free Press; 1995.
- ; World Health Organization. Informatics and Telematics in Health. Present and Potential Uses. Geneva: WHO; 1988.
|ICT: information and communication technology|
|NHS: National Health Service|
|SDO: Service Delivery and Organisation|
Edited by G. Eysenbach; submitted 18.08.05; peer-reviewed by C Pagliari, S Olsson; comments to author 15.09.05; revised version received 16.09.05; accepted 21.09.05; published 10.11.05
© Ray Jones, Ray Rogers, Jean Roberts, Lynne Callaghan, Laura Lindsey, John Campbell, Margaret Thorogood, Graham Wright, Nick Gaunt, Chris Hanks, Graham R Williamson. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 10.11.2005. Except where otherwise noted, articles published in the Journal of Medical Internet Research are distributed under the terms of the Creative Commons Attribution License (http://www.creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited, including full bibliographic details and the URL (see "please cite as" above), and this statement is included.