Published on in Vol 23, No 12 (2021): December

Preprints (earlier versions) of this paper are available at, first published .
Methods to Generate Innovative Research Ideas and Improve Patient and Public Involvement in Modern Epidemiological Research: Review, Patient Viewpoint, and Guidelines for Implementation of a Digital Cohort Study

Methods to Generate Innovative Research Ideas and Improve Patient and Public Involvement in Modern Epidemiological Research: Review, Patient Viewpoint, and Guidelines for Implementation of a Digital Cohort Study

Methods to Generate Innovative Research Ideas and Improve Patient and Public Involvement in Modern Epidemiological Research: Review, Patient Viewpoint, and Guidelines for Implementation of a Digital Cohort Study


1Deep Digital Phenotyping Research Unit, Department of Population Health, Luxembourg Institute of Health, Strassen, Luxembourg

2Diabetes Australia, Melbourne, Australia

3Diabetogenic, Melbourne, Australia

4Luxembourg Institute of Socio-Economic Research, Esch/Alzette, Luxembourg

5Centre of Research in Epidemiology and Statistic Sorbonne Paris Cité, National Institute of Health and Medical Research (INSERM), French National Institute for Agricultural Research (INRA), Université de Paris, Paris, France

6Centre d’Epidémiologie Clinique, Hôpital Hôtel-Dieu, Assistance Publique-Hôpitaux de Paris, Paris, France

7European Patients’ Academy on Therapeutic Innovation, Brussels, Belgium

Corresponding Author:

Gloria A Aguayo, MD, PhD

Deep Digital Phenotyping Research Unit, Department of Population Health

Luxembourg Institute of Health

1 AB rue Thomas Edison

Strassen, 1445


Phone: 352 26970770

Fax:352 269770719


Background: Patient and public involvement (PPI) in research aims to increase the quality and relevance of research by incorporating the perspective of those ultimately affected by the research. Despite these potential benefits, PPI is rarely included in epidemiology protocols.

Objective: The aim of this study is to provide an overview of methods used for PPI and offer practical recommendations for its efficient implementation in epidemiological research.

Methods: We conducted a review on PPI methods. We mirrored it with a patient advocate’s viewpoint about PPI. We then identified key steps to optimize PPI in epidemiological research based on our review and the viewpoint of the patient advocate, taking into account the identification of barriers to, and facilitators of, PPI. From these, we provided practical recommendations to launch a patient-centered cohort study. We used the implementation of a new digital cohort study as an exemplary use case.

Results: We analyzed data from 97 studies, of which 58 (60%) were performed in the United Kingdom. The most common methods were workshops (47/97, 48%); surveys (33/97, 34%); meetings, events, or conferences (28/97, 29%); focus groups (25/97, 26%); interviews (23/97, 24%); consensus techniques (8/97, 8%); James Lind Alliance consensus technique (7/97, 7%); social media analysis (6/97, 6%); and experience-based co-design (3/97, 3%). The viewpoint of a patient advocate showed a strong interest in participating in research. The most usual PPI modalities were research ideas (60/97, 62%), co-design (42/97, 43%), defining priorities (31/97, 32%), and participation in data analysis (25/97, 26%). We identified 9 general recommendations and 32 key PPI-related steps that can serve as guidelines to increase the relevance of epidemiological studies.

Conclusions: PPI is a project within a project that contributes to improving knowledge and increasing the relevance of research. PPI methods are mainly used for idea generation. On the basis of our review and case study, we recommend that PPI be included at an early stage and throughout the research cycle and that methods be combined for generation of new ideas. For e-cohorts, the use of digital tools is essential to scale up PPI. We encourage investigators to rely on our practical recommendations to extend PPI in future epidemiological studies.

J Med Internet Res 2021;23(12):e25743




Patient and public involvement (PPI) in research is defined as “research being carried out ‘with’ or ‘by’ members of the public rather than ‘to,’ ‘about,’ or ‘for’ them” [1]. PPI means that patients or the public are actively involved in the research process rather than being included only as participants. Public engagement is the 2-way process of engagement activities that benefits both the researcher and the public [2].

Research that involves patients and the public can reduce the mismatch between what matters to patients and what is actually being done in the research [3]. A waste of research resources can be generated when the needs of those likely to use the research results, such as patients and caregivers, are not taken into account [4]. PPI can contribute to the identification and selection of high-priority research questions, planning and performing of more focused research, and improvement of participants’ enrollment in clinical trials [5]. Ultimately, this can result in a higher societal benefit through better use of resources for research. PPI improves the quality of the study and makes research more relevant [6].

Involve, a UK-funded program, aims to improve the quality of research through the integration of PPI throughout the research cycle (identification and prioritization, commissioning, design and management, implementation, dissemination, implementation, and evaluation). Involve has published a report with guidelines to help researchers start new projects when they intend to include PPI in their projects [7].

Digital epidemiology has the same objectives as epidemiology, which are the observation of disease patterns, their evolution, and the causes of these patterns to improve population health and prevent diseases, but digital epidemiology uses digital data [8]. A digital or e-cohort study can integrate data that were not generated for the research (social media and registries) or were generated with digital tools (wearables, sensors, smartphone technologies, and e-questionnaires through web platforms) [9]. Before starting any epidemiological, clinical, or population-based study, researchers need to choose the best methodology to incorporate PPI throughout the project [10]. However, although there are approaches to integrate PPI in a research project, we think that there are no clear recommendations of which methods are the most appropriate, in particular with respect to the launch of cohort studies with digital sources of data.

Although PPI is recommended in research projects, this involvement is often not described or is incompletely reported [11]. A reason for underreporting may be to avoid describing an unsuccessful PPI attempt or that there was no involvement [12]. In addition, there is some evidence that PPI is seldom used in many countries [13]. In the case of cohort studies and, in particular, e-cohorts, we believe that this insufficient involvement may be due to a lack of knowledge of the methods, barriers, or facilitators to apply PPI. In addition, we think that there is a need to have concrete and clear examples for applying PPI in this type of study.


The aims of this study are to (1) review methodologies used to include PPI in research, (2) provide the viewpoint of a final user of research results, and (3) provide practical guidelines and recommendations about how to initiate and run an e-cohort study with PPI based on the review and the point of view of a patient advocate.

This work entails 3 parts: a narrative review about methodology and description of PPI, a viewpoint of a patient advocate, and a case study with practical guidelines and recommendations, illustrated by the implementation of a digital cohort study.


Data for this review were identified by searches of PubMed, Google Scholar, targeted websites about PPI, reports, and existing PPI guidelines, as well as Google Search and references from relevant studies. We used the following search terms: “patient and public participation,” “patient engagement,” “patient involvement,” “consumer involvement,” “community involvement,” “participatory health research,” “community based research,” “research ideas,” “co-writing,” “coproduction,” “co-design,” “cohort study,” “e-cohort,” and “longitudinal study.” We included original studies describing PPI methods using the Involve definition with at least one type of PPI in the study. We included information on studies published in English from 2000 to the present.

Viewpoint of a Patient Advocate

We used relevant definitions from the European Patients’ Academy on Therapeutic Innovation for patients. We defined patients as individuals with personal experience of the disease, caregivers as individuals supporting a patient, and patient advocates as individuals representing large numbers of patients with a specific disease [14].

We invited a patient advocate to present her perspective and expectations regarding PPI in the context of diabetes research and the use of digital tools and, in particular, the use of social media.


We integrated the results of the review and the patient advocate’s viewpoint to identify practical guidelines on how to increase PPI in future epidemiological studies. We used the implementation of a digital cohort study as an exemplary case for testing and illustrating established guidelines for PPI [15]. In addition, we integrated in our recommendations the revised version of Guidance for Reporting Involvement of Patients and the Public (GRIPP2) checklist as an instrument to improve the quality of PPI reporting [16].

This study was based on a collaboration of patients and researchers. A patient advocate (RS), patient researchers (TS and TB), and researchers (GAA, CG, AF, VTT, PR, LH, and GF) were involved in the preparation of the manuscript (cowriting, editing, and critical review).

Review of PPI Methods

We analyzed data from 97 studies published from 2000 to 2020. The studies were performed in 9 countries: the United Kingdom (58/97, 60%), Canada (13/97, 13%), the United States (8/97, 8%), Australia (6/97, 6%), Ireland (6/97, 6%), Denmark (2/97, 2%), France (1/97, 1%), Portugal (1/97, 1%), Indonesia (1/97, 1%), and the United Kingdom and Australia (1/97, 1%). The most frequent methods of PPI were workshops (47/97, 48%); surveys (33/97, 34%); meetings, events, or conferences (28/97, 29%); focus groups (25/97, 26%); interviews (23/97, 24%); consensus techniques (8/97, 8%); James Lind Alliance consensus technique (7/97, 7%); social media analysis (6/97, 6%); and experience-based co-design (3/97, 3%). Of the 97 studies, 34 (35%) used only 1 method, whereas 30 (31%), 22 (23%), 8 (8%), and 3 (3%) used 2, 3, 4, and 5 methods, respectively. The use of ≥3 methods together was observed from 2017 onwards (Figure 1 and Multimedia Appendix 1 [17-114]).

Figure 1. Number of fields or areas (ie, 1 field=involved in research ideas and 2 fields=research ideas and co-design) in which patients, carers, or the public were involved (y-axis); number of methods (circles); and countries (colors) where the studies were performed from 2000 to 2020 (x-axis). Patient and public involvement increases over time and at different stages of involvement. The size of each circle represents the number of methods used for patient and public involvement. Circles representing a combination of methods are very common in recently published studies. The most represented country is the United Kingdom.
View this figure

Format of Involvement

Broad Definition of PPI Used

When considering methods of approaching PPI, 2 situations should be distinguished. The first situation is where patients or the public provide data on ideas or research priorities. The second situation is where patients or the public are actively involved in decisions about the research and perform or collaborate in the design, data analysis, data interpretation, writing, or diffusion of results. For this review, we used a broad definition of PPI that includes both situations: patients or the public involved in generating research ideas and patients or the public actively involved in the research cycle [115].


We found that the most frequent method for approaching PPI was a workshop. Workshops are group activities where participants discuss a defined topic and decide actions. There are different ways of organizing a workshop. The first way is to organize it always in the same place and at the same hour. Morris et al [116] emphasized the importance of having enough space and time for the discussion. In addition, the number of participants by workshop or by discussion group may be limited. Mackintosh et al [17] proposed separate groups of users and service providers of 3-4 people. In contrast, Kelemen et al [18] organized full-day workshops with 20 to 25 participants of different origins and ages and divided into 2 groups randomly assigned. Patients prefer places with quiet rooms and parking lots, with meeting and meal times that allow them to participate and manage their medications [19]. An alternative to a workshop held at a specific location is to organize mobile workshops to facilitate the participation of people with busy schedules or mobility issues. Eccles et al [20] organized this type of workshop with a maximum of 10 participants and a short duration (30 minutes). These workshops were usually offered on the sites of existing organizations, which improved the success rate [20]. When face-to-face workshops are not possible, a workshop by videoconference can be organized [21]. Workshops can also be organized as consensus meetings [15,22], with a second-round workshop [117] or by applying other techniques to improve explicit involvement, such as experience-based co-design [23]. Workshops are usually analyzed with qualitative research methods such as thematic analysis [118] (Multimedia Appendix 1).


Surveys aim to obtain information in an easy, individual, and feasible way. They can include open-ended questions. For example, a citizen science study asked participants the magic wand question (“If you had a magic wand, what would you change in your healthcare?”) [24]. Surveys can also include closed-ended questions. Little et al [25] performed a study to identify patients’ preferences for a patient-centered consultation using Likert scale questions that were previously tested and validated in a pilot study. To respond to a survey, patients and the public could be contacted in person [25], through a web-based survey [119], or by telephone [120]. Several studies included a second round of questions for improving ideas or establishing priorities [26,121].

Focus Groups

Focus groups are group activities, a form of qualitative research that includes people with certain similarities such as a therapeutic area or illness or demographic or socioeconomic group to discuss their opinions and beliefs about a topic familiar to the participants [27]. An advantage of focus groups is the potential to obtain multiple opinions and different understandings of the same question, which can enrich explanations for a certain issue. A disadvantage is that focus groups require a trained interviewer to conduct them, and interaction among participants can influence the outcome [122].


Meetings are group activities where patients, carers, caregivers, and researchers participate in the project discussing and co-organizing activities for PPI [15]. They are complementary to workshops or focus groups and are also used to prepare these activities and to debrief methods or actions for PPI [28]. Meetings can be organized as festivals and public events, conferences [29], or web-based activities [30].

Semistructured Interviews

Semistructured interviews are a qualitative research method where the researcher discusses a defined topic individually and informally with the participant [123]. Using this method, nonresponse or direct behavioral observations, individual reflections on a specific research question, can be incorporated as outcomes; the advantage is that the participant is not influenced by other persons [124,125].

Consensus Techniques

Consensus techniques analyze a project’s chances of success by bringing together a group of experts at a workshop, meeting, or conference to discuss solutions and reach agreements [31]. James Lind Alliance is a consensus technique bringing clinicians, patients, carers, and the public together to achieve the convergence of opinions for establishing research priorities. The process starts with a web-based survey to collect research ideas from the public. Next, only unanswered research questions are selected. With a second web-based survey, the public prioritizes the selected questions. At the end of the process, the steering committee chooses the top 10 research priorities [126].


Persona-scenario is a method for co-design in PPI where a fictitious user (persona) is created to communicate in a committed manner. Next, a scenario is proposed based on a story with an actor (the persona), a framework, a goal, actions to reach this goal, and obstacles. Participants can be asked to evaluate the extent to which they agree with the choice of the persona or what they would do if they were in the situation of the persona. This distancing—giving advice about someone else’s choice rather than answering for yourself—can help address sensitive themes [23,32].

Experience-Based Co-design

Experience-based co-design is a technique using narratives, usually video recorded, that allows patients and researchers to work in partnership to co-design new services and technology with the objective of improving the quality of health services. This technique has been applied with success for fostering PPI [33-36].

Digital Methods to Promote PPI

Digital methods can be enablers of PPI, and the public can help modulate and develop digital technology [127]. Dedicated websites can enhance PPI not only by providing information, but also by organizing PPI itself. An example is the use of a webpage to organize a patient-led research hub. This is an initiative aimed at making patients and the public leaders of their own research projects. The researchers assist and support the patients and the public who proposed projects [37].

Crowdsourcing is a method by which many people are engaged on the web for a common goal such as obtaining new ideas and analyzing data. The main advantage is that it is possible to have a large number of contributors in a very short time [128]. However, because crowdsourcing participants are generally younger than participants from traditional nonweb-based procedures, these participants are not always representative of the target population [129].

Social media analysis is the use of digital data from social networks for epidemiological purposes. This information increases knowledge about epidemiological trends and may be very different from information obtained by traditional methods [130]. Social network data sources can be used, for example, to obtain information about the patient’s research priorities [38]. This often generates large quantities of data and can offer the opportunity to use special techniques such as natural language processing for the analysis [131].

Web-based platforms are internet services where the public and innovators meet and can be created for allowing coworking [132]. Vasilica et al [39] reported creating a web-based network to co-design a social media–based platform with the aim of generating information to improve disease outcomes [39]. In addition, web-based platforms through a web-based voting system have been used for establishing research priorities [133].

Fields of Involvement

We found that many studies (60/97, 62%) involved patients and the public for generation of new ideas. Patients or the public contributed also in coproduction, co-design, or study scope (42/97, 43%) and in establishing research priorities (31/97, 32%). Other forms of PPI included participation in data analysis (25/97, 26%), as coauthors of a scientific article (17/97, 18%), as members of a steering committee or advisory group (16/97, 16%), reviewing or writing protocols (14/97, 14%), in the interpretation of results (12/97, 12%), in the dissemination of results and advocacy (11/97, 11%), in the data collection (9/97, 9%), in the development of the recruitment strategy (7/97, 7%), as a project manager (4/97, 4%), and being coinvestigator or having patient- or public-led projects (2/97, 2%). We found only 1 study where patients were involved in the co-design of mobile health tools (1/97, 1%). Finally, recent studies reported an early PPI in at least three different stages or fields of involvement and throughout the life of the research project (Table 1 and Figure 1).

Table 1. Fields of involvement in the included studies (N=97).
Type of involvementFrequency, n (%)References
Generation of new ideas60 (62)[18,20,22-27,30,32-36,38-82,134]
Coproduction, co-design, or study scope42 (43)[17,19,23,30,32-37,39,41,49,54,55,59,60,65-69,74,75,78,81,83-98]
Establishing research priorities31 (32)[24,26,29,30,34,35,41-43,46-48,53,54,57,60,62-64,66,72,75,77,81,83,89,90,99-102]
Participation in data analysis25 (26)[24,30,31,34,51,60,61,66,69-71,78,81,87,90,91,94,96,103-109]
As coauthors of a scientific article17 (18)[24,30,31,57,60,61,68,69,73,83,86,94-96,105,110,111]
As members of a steering committee or advisory group16 (16)[26,28,30,31,35,42,43,53,62,76,78,84,91,106,110,112]
Reviewing or writing protocols14 (14)[30,49,51,59,60,68,73,74,78,83,91,95,111,113]
Interpretation of results12 (12)[24,30,41,49,60,66,69,78,91,94,96,104]
Dissemination of results and advocacy11 (11)[30,39,41,49,57,60,70,78,86,94,104]
Data collection9 (9)[54,60,66,70,88,91,94,96,114]
Development of the recruitment strategy7 (7)[41,60,66,68,91,104,111]
Project management4 (4)[30,37,78,94]
Coinvestigator or having patient- or public-led projects2 (2)[37,94]
Co-design of mobile health tools1 (1)[135]
At least three different stages or fields of involvement and throughout the life of the research project39 (40)[24,26,28,30,31,34,35,39,41-43,49,51,53,54,57,59-61,66,68-70,73-75, 78,81,83,86,89-91,94-96,104,110,111]

To develop a successful PPI project, patients or the public and researchers must have, or develop, certain skills. For example, researchers need to become familiar with PPI as a research approach, know how to manage a PPI project, and how to deal with conflict. As for the patients and the public, they must understand the research process and develop capacities for management and conflict management. However, it is not mandatory that patients have specific vocational or educational training [136].

PPI in Epidemiology: Trials and Cohort Studies

Web-based trials are more and more frequently described in the literature. Price et al [137] performed a systematic review of web-based trials and found that PPI was only reported in 24% (10/41) of the trials included in the review. Face-to-face meetings and email contact were the most common ways of interaction [137].

Taylor et al [97] performed a cohort study in patients with cancer that involved the patients in the creation and choice of a brand for the cohort. With a 1-day workshop, patients and researchers co-designed the brand. The results showed higher acceptance and retention of the study than expected. An ongoing cohort study used social media (Facebook) for PPI by creating a closed group of patients and families to bring new ideas to the project [80]. Meetings with a family advisory committee were organized regularly.

Morris et al [116], in the context of an epidemiological study, investigated PPI with surveys and postevent interviews and wrote recommendations. Before an event, they suggest sending a detailed document with the topics to be discussed. During the event, they suggest having enough space between tables to allow all participants to be heard, providing materials to facilitate note taking, taking a whole day for the meeting, and arranging a facilitator for each table. Finally, after the event, they recommend a follow-up by sending the notes to the participants [116].

Barriers to PPI

Domecq et al [138] in a systematic review described some barriers to PPI. They highlighted 2 barriers: the excessive time taken for training activities and attendance and the risk of a tokenistic involvement. Another barrier reported in participants who were frail was frustration because of discontinuity in the involvement [139]. Barriers reported by researchers were concerns about the quality of research, ethical issues, lack of funding, failure of the PPI in the past, and not being convinced of the real need for PPI in the cohort [140]. Maccarthy et al [113] described communication issues as a key barrier to PPI. Researchers fear not being able to explain the project and not being able to engage patients and the public in the project; they also feel discomfort speaking about their experiences with patients and the public and fear having misunderstandings.

Facilitators of PPI

Creating a safe and welcoming environment where each contributor feels empowered and confident facilitates PPI [94]. The coproduction process can give participants the self-confidence to take responsibility for the entire duration of a project [78]. In addition, an iterative process of PPI evaluation during the entire research cycle has been proposed to ensure success in PPI [113]. Mathie et al [141] reported that feedback for patients, when provided, motivated them to continue their collaboration with researchers. Concerns for well-being, trust, mutual respect, and flexibility in time and methods were facilitators of PPI [139]. Chambers et al [142] found key areas that may facilitate or hinder the development of PPI. These key areas were the following: good role definition, recognition of difficulties, integration through organizations, training, developing networking, considering different perspectives, improving communication, and recognizing the relevance of emotional impact. Finally, concerning digital interventions and PPI, O’Connor et al [143] recommend investing in raising awareness of the usefulness of digital tools, improving health literacy, and using optimal tool design.

A Patient Advocate’s Viewpoint of PPI in Diabetes Research

When writing this review, it was natural to allow a potential representative of study participants to express how they see PPI so far in research and what they are expecting from researchers to increase the participation of people with diabetes in research (Textbox 1, written by RS).

Involving people with diabetes in research: a patient advocate’s viewpoint.

Viewpoint of a patient advocate

  • On November 14 each year, World Diabetes Day is celebrated across the globe. The International Diabetes Federation with the World Health Organization created this awareness campaign in 1991 to respond to the growing numbers of people with diabetes worldwide. In 2006, World Diabetes Day was deemed an official United Nations Day with a special resolution, becoming only the second health condition so recognized.
  • Why November 14? That date marks the birthday of Sir Frederick Banting, who, along with Charles Best, is credited with the discovery of insulin. The day is acknowledged by diabetes organizations, health care professionals, researchers, and governments. Most importantly, people with diabetes have embraced the day to acknowledge, commemorate, and also celebrate life with diabetes as we gratefully signpost the man whose research is responsible for our very lives.
  • People with diabetes are interested in research. We know that the developments we see each and every year that advance how we live with diabetes are the result of research. We are interested in the different branches of research—clinical, educational, social, and behavioral—because we know better than anyone that living with diabetes is a multi-pronged existence that affects every part of our lives.
  • However, despite how much we appreciate the work of researchers and how keen we are to learn more, sometimes it is difficult to engage with us and involve us as participants. Let us explore how we can address this gap and consider some changes that can be made to encourage people with diabetes to take more interest in research.

Tell the story

  • The story of Banting and Best is folklore for those of us living with diabetes. It is a compelling story, but so are many other research tales. Unfortunately, the narrative is not always told especially effectively. It is difficult to make research sound relevant to people living each day with diabetes when research involves cells in a petri dish or stem cells in a temperature-controlled laboratory. What is the difference this work will make to our day-to-day lives?
  • It is exhausting for us to hear how mice are cured of diabetes (once again), especially when we know that our cure is still as elusive as ever.
  • However, these stories—the cells and the mice—are links in a long chain that lead to significant developments that do directly affect us. At the moment, explaining that seems lost in translation, and researchers need to think about how to decode in basic language how the work they are doing has the potential to make significant changes to the everyday life of people with diabetes and that participating in relevant research gets us to that goal.
  • Even those researchers whose work is more practical based are not always especially successful in describing the impact of their work on those of us with diabetes. Plain language statements are a start, but looking for even more nuanced and targeted ways to communicate is important.

Tell it in a tweet!

  • With 280 characters on offer, Twitter is the perfect platform for researchers to hone their short story–telling skills. Practice the elevator pitch of your research by narrowing down the key points and benefit to people with diabetes, and share it on the web to encourage interest. (Twitter threads allow for linking a number of tweets together, so if you need more than 280 characters, you can take a couple of tweets. But do keep it brief!)

What is involved?

  • When recruiting people for your research, be very clear about what they will need to do. How much time is involved? Where will they need to go? Will there be any invasive procedures and how uncomfortable are they likely to be? (Be honest!)
  • Follow-up is critically important. A complaint we hear from people participating in research is that once their involvement is over, they never again hear from the research team. This can be especially frustrating if people have invested a lot of time and energy in a trial. Regular updates through a newsletter or social media page keep people informed and linked in with your work. This is especially important if you are planning to recruit people for future phases of your study.

Participants, not subjects #LanguageMatters

  • The words you use when communicating to, and about, people with diabetes are critically important. Refer to language position statements developed by diabetes organizations to ensure that your language is supportive, empowering, positive, and encouraging. We people with diabetes are more inclined to be involved if we see a study that treats us with respect.

The 2 camps—and how to bridge them

  • There seem to be 2 main camps when it comes to diabetes research. Some believe that the focus should be primarily on finding a cure for diabetes. This seems to be especially prevalent in the type 1 diabetes space, with much of this thinking led by parents of children living with diabetes. In 1970, it was these parents who founded the leading diabetes research organization, the Juvenile Diabetes Foundation (renamed the Juvenile Diabetes Research Foundation), now known as JDRF. The original organization’s mission was very clear: to find a cure for diabetes. In recent years, however, the research funded by JDRF has branched out to include studies looking at improving management through technology and drugs.
  • However, it is important to acknowledge the importance of research that looks at better management. Without this research, there would be no treatment for diabetes-related complications and we would not have technology such as home blood glucose meters, insulin pumps, continuous and flash glucose monitors, algorithms to automate insulin delivery...and we would still be using the same insulin from dogs that Banting and Best had used.
  • Just as important is the growing body of work and researchers dedicated to researching the social, psychosocial, behavioral, and emotional aspects of living with diabetes. As anyone living with diabetes will tell you, this condition is never just about metrics. It is very much about our headspace and how we feel about living with diabetes.

Involve us

  • When is the best time to start to involve people with diabetes in your research? It is probably already too late! Have you consulted us when you were establishing your study design? And back up a little the research really something that is going to be of interest or benefit to people with diabetes. Is the problem you are looking to solve really a problem for us?

Patient advisory committees

  • Many research bodies now require patient advisory committees to be established as part of the overall study. Done well, these groups can provide invaluable input for projects. Done badly, they are nothing more than a tick-the-box exercise. Ensure that there is funding available for travel, accommodation, and other expenses. Be clear about what you expect the patient advisory committee members to do and which aspects of the project they will be involved in. Remember that patient advisory committee members will most likely be volunteering their time. Their expertise and time should be reimbursed by honoraria or hourly payments.

Not enough money in the pot

  • Research dollars are never enough, and each year, there are more researchers contending for elusive grants. When the results from grant rounds are shared, it seems that diabetes is repeatedly the poor cousin of health conditions, regularly being awarded significantly less money (with fewer successful grants) than conditions such as cancer and cardiovascular disease. In recent grant announcements from the National Health and Medical Research Council in Australia, only 16 diabetes grants worth Aus $13.5 million (US $9.6 million) were awarded compared with 69 grants worth Aus $52.9 million (US $37.8 million) awarded for cancer research.
  • People with diabetes can help to advance the cause of diabetes grant applications by telling their stories. Perhaps one of the reasons that diabetes receives comparatively little of the research bucket is because we have not been all that successful in telling our stories. Instead, we have created a false image of diabetes as a hugely preventable, self-inflicted condition, resulting in government and other research bodies considering diabetes a less worthy condition to fund.
  • Researchers are encouraged to work closely with people with diabetes to help tell the story of why their own research is important and how it has the potential to help in the lives of people affected by, or at risk for, diabetes. Humanizing the story is important—all too often, diabetes is presented in the media as a headless overweight body, which only adds to the stigma and image problem of the condition.

The story of hope

  • Research is selling an important feature: hope. People with diabetes trade on hope; we look for it in research because we know that is what holds the key to improving outcomes, reducing burden, and making our diabetes lives easier. We want to be part of those discoveries that promise a better life, and we want to be involved in your research that will help us get there.
Textbox 1. Involving people with diabetes in research: a patient advocate’s viewpoint.

Illustration of PPI in a Digital Cohort Study

Digital cohort studies are longitudinal studies in which the data come either totally (e-cohort) or partially (hybrid: e-cohort and traditional cohort) from digital sources. Modern cohort studies increasingly incorporate digital tools such as data generated on the web and connected devices that allow much wider use of data generated for multiple projects [9].

We elaborated a strategy of PPI for a digital cohort study. Tables 2 and 3 show recommendations for PPI at all stages of research. This participation was defined in 2 categories: Recommended participation activities, in which patients and the public participate more passively, helping to generate research ideas and prioritize those ideas by participating in surveys, and Recommended involvement activities, in which patients actively participate in collaborative work with researchers on an equal footing. Examples of these activities are events, meetings, workshops, and focus groups. The chosen strategies for the digital cohort study are detailed in 32 actions in total for participation as well as participation and involvement, corresponding to the different steps of the research process, and are based on the current recommendations of Involve [7], our review results, and the viewpoint of a patient advocate (Textbox 1). In addition, for the realization of certain activities, we suggest a recommended duration of the activity based on our review of the literature and the point of view of the patient advocate.

Table 2. Recommendations for the promotion of patient and public involvement (PPI) projects: concrete examples for a digital cohort study. Steps to be taken before starting the cohort study.
Stages (Involve list)Suggested actions

Recommended participation activitiesRecommended involvement activities
Identifying and prioritizing research axes
  • Web-based survey through social media: identification of research questions. Duration of the activity: 15 minutes
  • Videoconference meeting: establishing an international scientific steering committee with researchers and patients as members. Duration of the activity: 1 hour. Preparation: read agenda that should be sent 1 day before the meeting
  • Meeting: identification of, and invitation to, a group of patients interested to be involved as patient partners (eg, through patient associations). Duration of the activity: 3 hours. Preparation: not needed. Venue: local patient association
  • Videoconference meeting and use of web-based collaboration tools: cowriting PPI plan for the cohort and submission to an ethics committee. Duration of the activity: 2 hours. Preparation: read the proposal draft sent and written by researchers 1 week before
  • Web-based survey through smartphone app: identification of research questions, web-based survey with open-ended questions. Duration of the activity: 10 minutes
  • Web-based survey: ranking research questions (through smartphone app, web-based survey using persona-scenario technique) with closed-ended questions. Duration of the activity: 15 minutes
  • Web-based or mobile workshops: coproduction by giving feedback on study design and chosen questionnaires and research tools (such as mock-ups of app, user experience and user interface). Duration of the activity: 3 hours
  • Web-based training: language matters. Searching, choosing, and checking the most appropriate use of language for communication with the public and patients. Duration of the activity: 2 hours
  • Mobile focus group and survey (Multimedia Appendices 2 and 3): assessment of the beta version of smartphone app and flyers, as well as assessment of the wording and visual of the website, flyer, study objectives, and PPI expectations. Duration of the activity: 4 hours. Meeting place: comfortable, with catering and parking lot available
Drafting grant protocola
  • Web-based meeting: cowriting study protocol, involvement of patient associations as partners in grants. Duration of the activity: 1 hour. Preparation: read agenda that should be sent 1 day before the meeting
Testing and scaling up
  • Surveys through smartphone app: testing of pilot study by limited number of potential study participants. Duration of the activity: 30 minutes
  • Web-based meeting: co-designing pilot study on smartphone app. Duration of the activity: 1 hour. Preparation: read agenda that should be sent 1 day before the meeting
  • Web-based meeting: co-design of generalization phase and recruitment. Duration of the activity: 1 hour. Preparation: read agenda that should be sent 1 day before the meeting
  • Web-based meeting: advertise through social media and press for patients and the public to participate in the study

aNo specific recommendations.

Table 3. Recommendations for the promotion of patient and public involvement (PPI) projects: concrete examples for a digital cohort study. Steps to be taken during the cohort study.
Stages (Involve list)Suggested actions

Recommended participation activitiesRecommended involvement activities
Analyzing and interpretinga
  • Web-based meeting using web-based collaboration tools: cowriting of annual reports
  • Webpage, web-based workshops, meetings, and web-based collaboration tools: coproduction of research projects through a patient-led research hub. Webpage with a dedicated section for submission of projects by patient. Projects assessed by the scientific steering committee
  • Web-based workshops, meetings, and web-based collaboration tools: data analysis and interpretation of results
  • Meetings and web-based collaboration tools: writing of manuscripts cowritten by scientists and patients
  • Social media: dissemination of publications coauthored by scientists and patients
  • Web-based meetings, workshops, and web-based collaboration tools: participation at conferences as author or coauthor
  • Focus groups and workshops: communication of research results (plain language, infographic, and dissemination)
  • Web-based meetings and workshops: implementation of some results from the study at hospitals and consultations facilitated by patients
Monitoring and evaluating
  • Smartphone app: improving participants’ retention by reminders
  • Email newsletter and social media announcement: follow-up of the project by researchers (once a month)
  • One-day general public event: follow-up of the project by researchers (once a year)
  • Web-based survey through social media and smartphone app: monitoring evolution of the research protocol (adding or deleting research questions)
  • Web-based meetings: checking of data collection and data quality by patients
  • Web-based meetings and web-based collaboration tools: review of research projects by patient representatives
  • Social media: improving participants’ retention by involved patients
  • One-day general public event: follow-up of the project by patients and public (once a year). Remuneration or facilities for attending should be budgeted for members of the scientific committee
  • Workshop: evaluation of PPI by researcher and contributors (use GRIPP2b checklist). Duration of the activity: 1 day. Meeting place: comfortable with catering and available parking places
  • Meetings and workshops: Monitoring evolution of the research protocol (adding or deleting research questions) by scientific steering committee (scientists and patient members)

aNo specific recommendations.

bGRIPP2: revised version of Guidance for Reporting Involvement of Patients and the Public.

The advantage of a digital cohort study is that digital tools can be used to promote PPI at each stage as a primary method or as a complementary or alternative method. A digital cohort can facilitate PPI, allowing participation from remote locations, using a smartphone app with web-based questionnaires, organizing most of the meetings through videoconferences, and using web-based tools for coworking. However, we think that face-to-face activities are also recommended and these 2 approaches may be complementary.

The recommendations are as follows:

  • Recommendation 1: Identify patients and the public who might be interested in participating as members of the patient advisory steering committee. This contact can be achieved through social media and patients’ organizations. Use digital tools to identify people. For example, contact organizations that are active on forums or social media.
  • Recommendation 2: Write a PPI protocol in the digital cohort protocol describing all planned activities and include funding for patients. The patients who are part of the steering committee should be actively involved in this activity.
  • Recommendation 3: Identify patients and the public who might be interested in participating in focus groups, semistructured interviews, or workshops by being involved actively in the design of the app, website, research ideas, and project evaluation.
  • Recommendation 4: Organize focus groups as an important activity to obtain information about how PPI can be integrated in a digital cohort study. We present an example guide for this activity in Multimedia Appendix 2. A passionate and enthusiast moderator is needed. Sometimes it is not possible to find a place and a time that works for everyone. In such cases, mobile or web-based focus groups can be organized.
  • Recommendation 5: Involve patients in the design of the smartphone app or other digital tools. For working on the design of a smartphone app, we encourage the use of an evaluation grid to assess the app with direct observation of how patients and the public use it. We present an example of an evaluation grid in Multimedia Appendix 3. Patients and the public can also assess a smartphone app using a validated scale. We suggest the application of the user version of the Mobile Application Rating Scale [144].
  • Recommendation 6: Involve patients in the generation of new research ideas and research priorities. Using a smartphone app or through a website, we recommend that during the study, participants be invited to propose research questions at any time and as often as they wish. The patient advisory steering committee is involved in all the discussions about research priorities and follow-up of the project.
  • Recommendation 7: Give feedback to patients and the public through web-based newsletters and social media. Evaluate regularly satisfaction with PPI. To assess the impact of PPI among active participants and to define roles and changes in the project, we advise that annual workshops should be organized. At these annual workshops, PPI can also be assessed by contributors (researchers, patients, and the public) using the GRIPP2 checklist [16].
  • Recommendation 8: Establish a patient-researcher partnership. On the cohort’s website and in the smartphone app, we encourage the creation of a patient- or public-led research hub space. This space can receive applications from patients or the public to develop research projects. The scientific steering committee assesses applications concerning scientific content and feasibility. The research team helps patients and the public to carry out their own projects by providing methodological support.
  • Recommendation 9: Finally, we recommend inviting patient advocates, patients, or caregivers to become study ambassadors. They are volunteers actively involved in the study and social networks. The role of the study ambassadors will be to actively disseminate the results of the study and invite their networks to join the project.

Figure 2 shows our vision of integrating PPI in a digital cohort study in the whole research cycle.

Figure 2. Patient and public involvement in the research cycle of a digital cohort study. Digital tools are integrated at each stage of the research cycle, and some examples of digital tools are shown in the figure.
View this figure

Principal Findings

We observed that the most popular methods for PPI are workshops, focus groups, interviews, and surveys. The appropriate method must be carefully chosen to fit the research objective. For example, workshops and focus groups can be a rich source of information in the prestudy phase; they can be adapted to the context of the participants; and they allow direct observation of, and interaction among, participants. The downside is that they are highly dependent on the capacity of the moderators.

We have also described digital tools to collect information from the patients and the public for research, such as social media listening and a particular web-based crowdsourcing survey with the magic wand–like question [24]. We think that digital tools may not only facilitate patient engagement in research, but can also stimulate a continuous and long-term participation and involvement of patients and the public.

Overall, none of these methods can address all relevant questions for scientists and patients or the public related to a research project. Therefore, we recommend a mix of methods to obtain optimal and meaningful information for the research project at stake. Different methods can be applied at different times of the project depending on the objective. Regarding the timing of introducing PPI, we recommend that PPI be included from the beginning (eg, co-design and research plan) to the end (publication and dissemination), followed by a PPI assessment.

We suggest pragmatic steps to integrate PPI in future epidemiological studies. These recommendations are as generic as possible but may not be applicable in all cases. They are based on current guidelines for PPI with concrete examples for a digital cohort study [14,15]. These recommendations should be understood as modular, meaning that they must be adapted to the study design, population, available budget, duration of the project, and local context. For PPI report and assessment, we recommend using the GRIPP2 checklist, a dedicated PPI reporting checklist [16].

Comparison With Prior Work

Domecq et al [138], in a systematic review, identified barriers to, and facilitators of, PPI. They found no evidence indicating which method was best for PPI. In our review, we found that a combination of methods is more common with greater involvement (patients included in many areas or across the research cycle). We integrated the barriers and facilitators described by Domecq et al [138] in our recommendations.

Liabo et al [145] performed a systematic review of good PPI practices. They compared the results of the systematic review with 3 involvement groups and found that the priorities were similar. However, the involvement groups found additional values that were not described in the literature, such as the enthusiasm of the participants and the choice of welcoming venues for the meetings. We integrated these reflections in our recommendations.

Nunn et al [146] reviewed reports of 96 human genomic research projects and found that only 33% (32/96) declared PPI. From these, most of the PPI activities were organized in formal groups (20/32, 63%), with 22% (7/32) using web-based tools (website, social media, and web-based communities). We found similar results with social media reporting in 6% (6/97) of the studies with PPI. We think that there is room for more PPI using digital tools.

Miah et al [147] conducted a scoping review of PPI in dementia research and found 19 studies from the United Kingdom and 1 from the Netherlands. Biddle et al [13] found an uneven distribution of PPI in Europe. They attributed this to a lack of infrastructure, support, and guidance. An example of support is that research-funding institutions in the United Kingdom require PPI in project applications. However, funding agencies in many countries do not have this requirement. We found similar results with most of the studies on PPI from the United Kingdom. We believe that PPI is still underreported or not performed in many countries.

Few epidemiological or clinical studies report PPI in the research process [11,146]. Studies that include patients and the public most often involve them only at the stage of idea generation, but not in the whole research cycle. This suggests that PPI is symbolically added or very limited.

Individual interviews are useful for tackling sensitive questions because participants do not feel dominated or influenced by the opinion of other participants. Surveys (web-based, telephone-based, or paper-based) are a pragmatic method of obtaining large-scale information from many people quickly. Differently, a survey can be very useful for generating new ideas and for refining and improving them when a new survey containing the generated ideas is launched for another sample of people.

PPI is a project in the project, creating its own challenges such as completing appropriate regulatory tasks and obtaining approval from an ethics committee. In the informed consent, the nature of participation must be specified, with clear and fair terms and conditions. This includes whether remuneration, cost coverage for the patient organization, and travel costs of patients or the public are provided and whether there are other benefits for the patients, such as nonfinancial compensation for the time allocated to project participation [14].

Several reasons for a lack of, or delay in, PPI have been mentioned in the literature. Some researchers do not include patients or the public in the research, arguing that the patient or public point of view is too subjective [148]. Furthermore, researchers may consider PPI only as a requirement of funding agencies for the project to be approved; therefore, PPI arrives late in the process and is treated as an afterthought [149]. We believe that obstacles can be overcome when they are identified and taken into account in the plan and recommendations.

When PPI is considered, there is the risk of selective PPI, which means that only those within the community who agreed with the research objectives are included. In addition, this may have the risk of hearing only the opinion of the most active patients or public, which may not reflect the opinion of most of the other patients and can therefore lead to research designs that still do not research the questions that interest most patients. We recommend nonselective PPI, meaning that the aim should be to involve a diverse selection of participants in the PPI process, including patient organizations, patient advocates who are legitimately speaking on behalf of a patient community, and individual patients, to ensure that the opinions and views of the participants are representative of most patients with the same condition [47].

Researchers should make a greater effort to minimize the burden of participation for patients and maximize the benefits for participants at the same time. A lack of participation is more likely to occur when there is poor or 1-way communication, resulting in poorly organized protocols and demanding follow-ups that lead to noninformed, noninvolved, and nonmotivated participants. Nevertheless, when done properly, patients can associate their participation with feelings such as usefulness, empowerment, and consideration [150].


This study includes certain limitations. Our search was limited to studies in English. In addition, a part of what is happening in PPI is described in the gray literature and has therefore been excluded from the review. Our review may give the impression that early involvement in setting research priorities is the norm, but this may not be true because of a potential publication bias. For example, the results of studies with PPI are often not intended to show improvements in efficacy in clinical studies; therefore, they are less likely to be accepted by publishers for publication or publication may be delayed because of negative or statistically insignificant results.


There are, and rightly so, many expectations on the part of patients and the public to be actively involved in research and not only by providing data, but also as research partners. PPI can contribute to patient empowerment by increasing disease awareness and according recognition as actors of their own condition [151]. However, PPI is uneven among countries and research institutions, and even now many patients and the public are not yet involved in research [13] and ignore or do not have access to research protocols [152]. Digital tools such as websites, social media, and connected devices have been increasingly incorporated into cohort studies and could be leveraged to increase PPI [153]. Digital tools can facilitate PPI by providing an opportunity for remote access and therefore easier participation. In addition, digital tools can facilitate PPI by enabling feedback and interaction between researchers and patient collaborators [154].

PPI can be a powerful approach to increase the relevance of research projects. We have shown that PPI must be planned in the initial phases of the development of a new epidemiological study and then be considered throughout the life of the research project. Combining different approaches of PPI seems to be the most effective strategy for improving the quality of research.

Some techniques such as persona-scenario are very powerful for idea generation and can be combined with digital tools. In addition, web-based surveys are easy to implement and allow involving many participants (crowdsourcing).

Digital methods such as social media listening or web-based magic wand–like questions can also offer useful complementary channels of interaction and help to identify key information such as research gaps at large scale with a limited cost directly from the target population. As such, we recommend that these methods be also integrated in the PPI process. With the example of a new digital cohort study, we offer practical guidelines to implement and run a patient- or public-centered research study. We therefore encourage investigators to rely on our practical recommendations to increase PPI in future epidemiological studies.


This study was funded by the Ministry of Higher Education and Research of Luxembourg and the Luxembourg Institute of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

The authors thank Ms Sandrine Lavalle for her support for this work.

Authors' Contributions

GAA reviewed the literature, selected the studies, extracted and analyzed data, and wrote the first draft. RS wrote Textbox 1 and contributed to the review and editing of the manuscript. CG, AF, TS, VTT, PR, TB, and LH contributed to the writing and editing of the manuscript. GF designed the study, selected and analyzed the data, and contributed to the writing and editing of the manuscript. All authors agreed on the final version of the manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Methods and fields of patient and public involvement.

DOCX File , 152 KB

Multimedia Appendix 2

A focus group guide.

DOCX File , 13 KB

Multimedia Appendix 3

Grid for assessment of a smartphone app.

DOCX File , 15 KB

  1. Tips sheet: recruiting members of the public to get involved in research funding and commissioning processes. INVOLVE - National Institute for Health Research. 2012.   URL: [accessed 2019-11-20]
  2. What is patient and public involvement and public engagement? National Institute for Health Research. 2020.   URL: [accessed 2021-12-07]
  3. Trish Greenhalgh: towards an institute for patient-led research. The BMJ Opinion. 2019.   URL: https:/​/blogs.​​bmj/​2019/​11/​12/​trisha-greenhalgh-towards-an-institute-for-patient-led-research/​ [accessed 2021-12-07]
  4. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet 2014 Jan 11;383(9912):156-165. [CrossRef] [Medline]
  5. Crocker JC, Ricci-Cabello I, Parker A, Hirst JA, Chant A, Petit-Zeman S, et al. Impact of patient and public involvement on enrolment and retention in clinical trials: systematic review and meta-analysis. Br Med J 2018 Nov 28;363:k4738 [FREE Full text] [CrossRef] [Medline]
  6. Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect 2014 Oct;17(5):637-650. [CrossRef] [Medline]
  7. Hayes H, Buckland S, Tarpey M. Briefing Notes for Researchers: Public Involvement in NHS, Public Health and Social Care Research. Eastleigh: INVOLVE; 2012.
  8. Salathé M. Digital epidemiology: what is it, and where is it going? Life Sci Soc Policy 2018 Jan 04;14(1):1 [FREE Full text] [CrossRef] [Medline]
  9. Fagherazzi G, Ravaud P. Digital diabetes: perspectives for diabetes prevention, management and research. Diabetes Metab 2019 Sep;45(4):322-329. [CrossRef] [Medline]
  10. Boote J, Baird W, Beecroft C. Public involvement at the design stage of primary health research: a narrative review of case examples. Health Policy 2010 Apr;95(1):10-23. [CrossRef] [Medline]
  11. Harris J, Haltbakk J, Dunning T, Austrheim G, Kirkevold M, Johnson M, et al. How patient and community involvement in diabetes research influences health outcomes: a realist review. Health Expect 2019 Oct;22(5):907-920 [FREE Full text] [CrossRef] [Medline]
  12. Price A, Schroter S, Snow R, Hicks M, Harmston R, Staniszewska S, et al. Frequency of reporting on patient and public involvement (PPI) in research studies published in a general medical journal: a descriptive study. BMJ Open 2018 Dec 23;8(3):e020452 [FREE Full text] [CrossRef] [Medline]
  13. Biddle MS, Gibson A, Evans D. Attitudes and approaches to patient and public involvement across Europe: a systematic review. Health Soc Care Community 2021 Jan;29(1):18-27. [CrossRef] [Medline]
  14. Hunter A, Facey K, Thomas V, Haerry D, Warner K, Klingmann I, et al. EUPATI guidance for patient involvement in medicines research and development: health technology assessment. Front Med (Lausanne) 2018 Sep 6;5:231 [FREE Full text] [CrossRef] [Medline]
  15. INVOLVE. Briefing Notes for Researchers: Public Involvement in NHS, Public Health and Social Care Research. Eastleigh: INVOLVE; 2012.
  16. Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. Res Involv Engagem 2017;3:13 [FREE Full text] [CrossRef] [Medline]
  17. Mackintosh N, Sandall J, Collison C, Carter W, Harris J. Employing the arts for knowledge production and translation: visualizing new possibilities for women speaking up about safety concerns in maternity. Health Expect 2018 Jun 17;21(3):647-658 [FREE Full text] [CrossRef] [Medline]
  18. Kelemen M, Surman E, Dikomitis L. Cultural animation in health research: an innovative methodology for patient and public involvement and engagement. Health Expect 2018 Aug;21(4):805-813 [FREE Full text] [CrossRef] [Medline]
  19. Evans BA, Porter A, Snooks H, Burholt V. A co-produced method to involve service users in research: the SUCCESS model. BMC Med Res Methodol 2019 Feb 15;19(1):34 [FREE Full text] [CrossRef] [Medline]
  20. Eccles A, Bryce C, Turk A, Atherton H. Patient and public involvement mobile workshops - convenient involvement for the un-usual suspects. Res Involv Engagem 2018;4:38 [FREE Full text] [CrossRef] [Medline]
  21. How do I hold a PPI meeting using virtual tools? National Institute for Health Research. 2020.   URL: https:/​/www.​​PPI/​resources-for-researchers/​faq/​how-do-i-hold-ppi-meetings-using-virtual-tools [accessed 2020-10-22]
  22. Boddy K, Cowan K, Gibson A, Britten N. Does funded research reflect the priorities of people living with type 1 diabetes? A secondary analysis of research questions. BMJ Open 2017 Sep 27;7(9):e016540 [FREE Full text] [CrossRef] [Medline]
  23. Knowles S, Hays R, Senra H, Bower P, Locock L, Protheroe J, et al. Empowering people to help speak up about safety in primary care: using codesign to involve patients and professionals in developing new interventions for patients with multimorbidity. Health Expect 2018 Apr;21(2):539-548 [FREE Full text] [CrossRef] [Medline]
  24. Tran V, Riveros C, Péan C, Czarnobroda A, Ravaud P. Patients' perspective on how to improve the care of people with chronic conditions in France: a citizen science study within the ComPaRe e-cohort. BMJ Qual Saf 2019 Nov;28(11):875-886 [FREE Full text] [CrossRef] [Medline]
  25. Little P, Everitt H, Williamson I, Warner G, Moore M, Gould C, et al. Preferences of patients for patient centred approach to consultation in primary care: observational study. Br Med J 2001 Feb 24;322(7284):468-472 [FREE Full text] [CrossRef] [Medline]
  26. Boney O, Bell M, Bell N, Conquest A, Cumbers M, Drake S, et al. Identifying research priorities in anaesthesia and perioperative care: final report of the joint National Institute of Academic Anaesthesia/James Lind Alliance Research Priority Setting Partnership. BMJ Open 2015 Dec 16;5(12):e010006 [FREE Full text] [CrossRef] [Medline]
  27. Higgins PS, Shugrue N, Ruiz K, Robison J. Medicare and Medicaid users speak out about their health care: the real, the ideal, and how to get there. Popul Health Manag 2015 Apr;18(2):123-130. [CrossRef] [Medline]
  28. Howe A, Mathie E, Munday D, Cowe M, Goodman C, Keenan J, et al. Learning to work together - lessons from a reflective analysis of a research project on public involvement. Res Involv Engagem 2017;3:1 [FREE Full text] [CrossRef] [Medline]
  29. Rawson TM, Castro-Sánchez E, Charani E, Husson F, Moore LS, Holmes AH, et al. Involving citizens in priority setting for public health research: implementation in infection research. Health Expect 2018 Feb;21(1):222-229 [FREE Full text] [CrossRef] [Medline]
  30. Dewa LH, Lawrence-Jones A, Crandell C, Jaques J, Pickles K, Lavelle M, et al. Reflections, impact and recommendations of a co-produced qualitative study with young people who have experience of mental health difficulties. Health Expect 2020 Jun 09. [CrossRef] [Medline]
  31. O'Hara MC, Hynes L, O'Donnell M, Keighron C, Allen G, Caulfield A, with the D1 Now Type 1 Diabetes Young Adult Study Group. Strength in Numbers: an international consensus conference to develop a novel approach to care delivery for young adults with type 1 diabetes, the D1 Now Study. Res Involv Engagem 2017;3:25 [FREE Full text] [CrossRef] [Medline]
  32. Valaitis R, Longaphy J, Ploeg J, Agarwal G, Oliver D, Nair K, et al. Health TAPESTRY: co-designing interprofessional primary care programs for older adults using the persona-scenario method. BMC Fam Pract 2019 Sep 04;20(1):122 [FREE Full text] [CrossRef] [Medline]
  33. Locock L, Robert G, Boaz A, Vougioukalou S, Shuldham C, Fielden J, et al. Using a national archive of patient experience narratives to promote local patient-centered quality improvement: an ethnographic process evaluation of 'accelerated' experience-based co-design. J Health Serv Res Policy 2014 Oct 19;19(4):200-207. [CrossRef] [Medline]
  34. Hatton AL, Haslam C, Bell S, Langley J, Woolrych R, Cory C, et al. Innovative solutions to enhance safe and green environments for ageing well using co-design through patient and public involvement. Res Involv Engagem 2020;6:45 [FREE Full text] [CrossRef] [Medline]
  35. Hyatt A, Morkunas B, Davey D, Thai AA, Trewhella M, Duffy M, et al. Co-design and development of online video resources about immunotherapy with patients and their family. Patient Educ Couns 2020 Sep 16. [CrossRef] [Medline]
  36. Mulvale G, Green J, Miatello A, Cassidy AE, Martens T. Finding harmony within dissonance: engaging patients, family/caregivers and service providers in research to fundamentally restructure relationships through integrative dynamics. Health Expect 2020 Jun 11. [CrossRef] [Medline]
  37. Mader LB, Harris T, Kläger S, Wilkinson IB, Hiemstra TF. Inverting the patient involvement paradigm: defining patient led research. Res Involv Engagem 2018 Jul;4:21 [FREE Full text] [CrossRef] [Medline]
  38. Priest C, Knopf A, Groves D, Carpenter JS, Furrey C, Krishnan A, et al. Finding the patient's voice using big data: analysis of users' health-related concerns in the ChaCha question-and-answer service (2009-2012). J Med Internet Res 2016;18(3):e44 [FREE Full text] [CrossRef] [Medline]
  39. Vasilica CM, Brettle A, Ormandy P. A co-designed social media intervention to satisfy information needs and improve outcomes of patients with chronic kidney disease: longitudinal study. JMIR Form Res 2020 Jan 27;4(1):e13207 [FREE Full text] [CrossRef] [Medline]
  40. Piercy H, Yeo M, Yap S, Hart AR. What are the information needs of parents caring for a child with Glutaric aciduria type 1? BMC Pediatr 2019 Oct 13;19(1):349 [FREE Full text] [CrossRef] [Medline]
  41. Synnot A, Bragge P, Lowe D, Nunn JS, O'Sullivan M, Horvat L, et al. Research priorities in health communication and participation: international survey of consumers and other stakeholders. BMJ Open 2018 May 08;8(5):e019481 [FREE Full text] [CrossRef] [Medline]
  42. Finer S, Robb P, Cowan K, Daly A, Shah K, Farmer A. Setting the top 10 research priorities to improve the health of people with Type 2 diabetes: a Diabetes UK-James Lind Alliance Priority Setting Partnership. Diabet Med 2018 Jul 15;35(7):862-870 [FREE Full text] [CrossRef] [Medline]
  43. Bethell J, Puts MT, Sattar S, Andrew MK, Choate AS, Clarke B, et al. The Canadian frailty priority setting partnership: research priorities for older adults living with frailty. Can Geriatr J 2019 Mar;22(1):23-33 [FREE Full text] [CrossRef] [Medline]
  44. Berwick DM, Loehrer S, Gunther-Murphy C. Breaking the rules for better care. J Am Med Assoc 2017 Jun 06;317(21):2161-2162. [CrossRef] [Medline]
  45. Zimbudzi E, Lo C, Robinson T, Ranasinha S, Teede HJ, Usherwood T, et al. The impact of an integrated diabetes and kidney service on patients, primary and specialist health professionals in Australia: a qualitative study. PLoS One 2019;14(7):e0219685 [FREE Full text] [CrossRef] [Medline]
  46. Gregg A, Getz N, Benger J, Anderson A. A novel collaborative approach to building better clinical trials: new insights from a patient engagement workshop to propel patient-centricity forward. Ther Innov Regul Sci 2019 May 22:2168479019849875. [CrossRef] [Medline]
  47. Russell G, Starr S, Elphick C, Rodogno R, Singh I. Selective patient and public involvement: the promise and perils of pharmaceutical intervention for autism. Health Expect 2018 Apr;21(2):466-473 [FREE Full text] [CrossRef] [Medline]
  48. Woodward L, Johnson S, Walle JV, Beck J, Gasteyger C, Licht C, aHUS Registry SAB. An innovative and collaborative partnership between patients with rare disease and industry-supported registries: the Global aHUS Registry. Orphanet J Rare Dis 2016 Nov 21;11(1):154 [FREE Full text] [CrossRef] [Medline]
  49. Westfall JM, VanVorst RF, Main DS, Herbert C. Community-based participatory research in practice-based research networks. Ann Fam Med 2006;4(1):8-14 [FREE Full text] [CrossRef] [Medline]
  50. Neville C, Da CD, Mill C, Rochon M, Aviña-Zubieta JA, Pineau CA, et al. The needs of persons with lupus and health care providers: a qualitative study aimed toward the development of the Lupus Interactive Navigator™. Lupus 2014 Feb;23(2):176-182. [CrossRef] [Medline]
  51. Jørgensen CR, Eskildsen NB, Johnsen AT. User involvement in a Danish project on the empowerment of cancer patients - experiences and early recommendations for further practice. Res Involv Engagem 2018;4:26 [FREE Full text] [CrossRef] [Medline]
  52. Casanova T, Black C, Rafiq S, Hugill-Jones J, Read JC, Vancleef K. The impact of active research involvement of young children in the design of a new stereotest. Res Involv Engagem 2020;6:29 [FREE Full text] [CrossRef] [Medline]
  53. Birnie KA, Dib K, Ouellette C, Dib MA, Nelson K, Pahtayken D, et al. Partnering for Pain: a Priority Setting Partnership to identify patient-oriented research priorities for pediatric chronic pain in Canada. CMAJ Open 2019;7(4):654-664 [FREE Full text] [CrossRef] [Medline]
  54. Brainard JS, Al Assaf E, Omasete J, Leach S, Hammer CC, Hunter PR. Forced migrants involved in setting the agenda and designing research to reduce impacts of complex emergencies: combining with patient and public involvement. Res Involv Engagem 2017;3:23 [FREE Full text] [CrossRef] [Medline]
  55. Brett L, Nguyen AD, Siette J, Dove-Pizarro J, Hourihan F, Georgiou A. The co-design of timely and meaningful information needed to enhance social participation in community aged care services: think tank proceedings. Australas J Ageing 2020 Mar;39(1):162-167. [CrossRef] [Medline]
  56. Brown K, Dyas J, Chahal P, Khalil Y, Riaz P, Cummings-Jones J. Discovering the research priorities of people with diabetes in a multicultural community: a focus group study. Br J Gen Pract 2006 Mar;56(524):206-213 [FREE Full text] [Medline]
  57. Costello W, Dorris E. Laying the groundwork: building relationships for public and patient involvement in pre-clinical paediatric research. Health Expect 2020 Feb;23(1):96-105 [FREE Full text] [CrossRef] [Medline]
  58. Coultas C, Kieslich K, Littlejohns P. Patient and public involvement in priority-setting decisions in England's Transforming NHS: an interview study with Clinical Commissioning Groups in South London sustainability transformation partnerships. Health Expect 2019 Dec;22(6):1223-1230 [FREE Full text] [CrossRef] [Medline]
  59. Coupe N, Mathieson A. Patient and public involvement in doctoral research: impact, resources and recommendations. Health Expect 2020 Feb;23(1):125-136 [FREE Full text] [CrossRef] [Medline]
  60. Dawson S, Ruddock A, Parmar V, Morris R, Cheraghi-Sohi S, Giles S, et al. Patient and public involvement in doctoral research: reflections and experiences of the PPI contributors and researcher. Res Involv Engagem 2020;6:23 [FREE Full text] [CrossRef] [Medline]
  61. Frost J, Gibson A, Harris-Golesworthy F, Harris J, Britten N. Patient involvement in qualitative data analysis in a trial of a patient-centred intervention: reconciling lay knowledge and scientific method. Health Expect 2018 Dec;21(6):1111-1121 [FREE Full text] [CrossRef] [Medline]
  62. Gadsby R, Snow R, Daly AC, Crowe S, Matyka K, Hall B, et al. Setting research priorities for Type 1 diabetes. Diabet Med 2012 Oct;29(10):1321-1326. [CrossRef] [Medline]
  63. Ghisoni M, Wilson CA, Morgan K, Edwards B, Simon N, Langley E, et al. Priority setting in research: user led mental health research. Res Involv Engagem 2017;3:4 [FREE Full text] [CrossRef] [Medline]
  64. Gillard S, Simons L, Turner K, Lucock M, Edwards C. Patient and public involvement in the coproduction of knowledge: reflection on the analysis of qualitative data in a mental health study. Qual Health Res 2012 Aug;22(8):1126-1137. [CrossRef] [Medline]
  65. Grant C, Widnall E, Cross L, Simonoff E, Downs J. Informing the development of an E-platform for monitoring wellbeing in schools: involving young people in a co-design process. Res Involv Engagem 2020;6:51 [FREE Full text] [CrossRef] [Medline]
  66. Grundy A, Keetharuth AD, Barber R, Carlton J, Connell J, Buck ET, et al. Public involvement in health outcomes research: lessons learnt from the development of the recovering quality of life (ReQoL) measures. Health Qual Life Outcomes 2019 Apr 11;17(1):60 [FREE Full text] [CrossRef] [Medline]
  67. Irving A, Turner J, Marsh M, Broadway-Parkinson A, Fall D, Coster J, et al. A coproduced patient and public event: an approach to developing and prioritizing ambulance performance measures. Health Expect 2018 Feb;21(1):230-238 [FREE Full text] [CrossRef] [Medline]
  68. Madden M, Morris S, Ogden M, Lewis D, Stewart D, McCambridge J. Producing co-production: reflections on the development of a complex intervention. Health Expect 2020 Jun;23(3):659-669 [FREE Full text] [CrossRef] [Medline]
  69. Mann C, Chilcott S, Plumb K, Brooks E, Man M. Reporting and appraising the context, process and impact of PPI on contributors, researchers and the trial during a randomised controlled trial - the 3D study. Res Involv Engagem 2018;4:15 [FREE Full text] [CrossRef] [Medline]
  70. Mathie E, Smeeton N, Munday D, Rhodes G, Wythe H, Jones J. The role of patient and public involvement leads in facilitating feedback: "invisible work". Res Involv Engagem 2020;6:40 [FREE Full text] [CrossRef] [Medline]
  71. McCarron TL, Noseworthy T, Moffat K, Wilkinson G, Zelinsky S, White D, et al. A co-designed framework to support and sustain patient and family engagement in health-care decision making. Health Expect 2020 Aug;23(4):825-836 [FREE Full text] [CrossRef] [Medline]
  72. Murta FR, Waxman J, Skilton A, Wickwar S, Bonstein K, Cable R, et al. The first UK national blepharospasm patient and public involvement day; identifying priorities. Orbit 2020 Aug;39(4):233-240. [CrossRef] [Medline]
  73. Ní Shé E, Cassidy J, Davies C, De Brún A, Donnelly S, Dorris E, et al. Minding the gap: identifying values to enable public and patient involvement at the pre-commencement stage of research projects. Res Involv Engagem 2020;6:46 [FREE Full text] [CrossRef] [Medline]
  74. Nissen ER, Bregnballe V, Mehlsen MY, Muldbjerg A, O'Connor M, Lomborg KE. Patient involvement in the development of a psychosocial cancer rehabilitation intervention: evaluation of a shared working group with patients and researchers. Res Involv Engagem 2018;4:24 [FREE Full text] [CrossRef] [Medline]
  75. O'Donnell D, Ní Shé E, McCarthy M, Thornton S, Doran T, Smith F, et al. Enabling public, patient and practitioner involvement in co-designing frailty pathways in the acute care setting. BMC Health Serv Res 2019 Nov 05;19(1):797 [FREE Full text] [CrossRef] [Medline]
  76. Porcheret M, Grime J, Main C, Dziedzic K. Developing a model osteoarthritis consultation: a Delphi consensus exercise. BMC Musculoskelet Disord 2013 Jan 16;14:25 [FREE Full text] [CrossRef] [Medline]
  77. Rankin G, Summers R, Cowan K, Barker K, Button K, Carroll SP, James Lind Alliance (JLA) Physiotherapy Priority Setting Partnership (PSP) Steering Group. Identifying priorities for physiotherapy research in the UK: the James Lind Alliance physiotherapy priority setting partnership. Physiotherapy 2020 Jun;107:161-168. [CrossRef] [Medline]
  78. Read S, Aries AM, Ashby SM, Bambrick V, Blackburn SJ, Clifford H, et al. Facilitating personal development for public involvement in health-care education and research: a co-produced pilot study in one UK higher education institute. Health Expect 2020 Oct;23(5):1191-1201 [FREE Full text] [CrossRef] [Medline]
  79. Robinson T, Skouteris H, Burns P, Melder A, Bailey C, Croft C, et al. Flipping the paradigm: a qualitative exploration of research translation centres in the United Kingdom and Australia. Health Res Policy Syst 2020 Sep 29;18(1):111 [FREE Full text] [CrossRef] [Medline]
  80. Walsh NM, O'Brien EC, Geraghty AA, Byrne DF, Whelan A, Reilly S, et al. Taking guidance from parents involved in a longitudinal birth cohort - the ROLO family advisory committee. Res Involv Engagem 2020;6:16 [FREE Full text] [CrossRef] [Medline]
  81. Wittmeier KD, Hobbs-Murison K, Holland C, Crawford E, Loewen H, Morris M, et al. Identifying information needs for hirschsprung disease through caregiver involvement via social media: a prioritization study and literature review. J Med Internet Res 2018 Dec 21;20(12):e297 [FREE Full text] [CrossRef] [Medline]
  82. Puppo C, Spire B, Morel S, Génin M, Béniguel L, Costagliola D, et al. How PrEP users constitute a community in the MSM population through their specific experience and management of stigmatization. The example of the French ANRS-PREVENIR study. AIDS Care 2020 May;32(sup2):32-39. [CrossRef] [Medline]
  83. Young HM, Goodliffe S, Madhani M, Phelps K, Regen E, Locke A, et al. Co-producing progression criteria for feasibility studies: a partnership between patient contributors, clinicians and researchers. Int J Environ Res Public Health 2019 Oct 06;16(19) [FREE Full text] [CrossRef] [Medline]
  84. Sin J, Henderson C, Woodham LA, Hernández A, Gillard S. A multicomponent eHealth intervention for family carers for people affected by psychosis: a coproduced design and build study. J Med Internet Res 2019 Aug 06;21(8):e14374 [FREE Full text] [CrossRef] [Medline]
  85. Bell T, Vat LE, McGavin C, Keller M, Getchell L, Rychtera A, et al. Co-building a patient-oriented research curriculum in Canada. Res Involv Engagem 2019 Feb 11;5(1):7 [FREE Full text] [CrossRef] [Medline]
  86. Tapp H, Derkowski D, Calvert M, Welch M, Spencer S. Patient perspectives on engagement in shared decision-making for asthma care. Fam Pract 2017 Jun 01;34(3):353-357 [FREE Full text] [CrossRef] [Medline]
  87. Locock L, Kirkpatrick S, Brading L, Sturmey G, Cornwell J, Churchill N, et al. Involving service users in the qualitative analysis of patient narratives to support healthcare quality improvement. Res Involv Engagem 2019;5:1 [FREE Full text] [CrossRef] [Medline]
  88. Buffel T. Social research and co-production with older people: developing age-friendly communities. J Aging Stud 2018 Mar;44:52-60 [FREE Full text] [CrossRef] [Medline]
  89. Carr EC, Patel JN, Ortiz MM, Miller JL, Teare SR, Barber CE, et al. Co-design of a patient experience survey for arthritis central intake: an example of meaningful patient engagement in healthcare design. BMC Health Serv Res 2019 Jun 04;19(1):355 [FREE Full text] [CrossRef] [Medline]
  90. Dahm MR, Brown A, Martin D, Williams M, Osborne B, Basseal J, et al. Interaction and innovation: practical strategies for inclusive consumer-driven research in health services. BMJ Open 2019 Dec 16;9(12):e031555 [FREE Full text] [CrossRef] [Medline]
  91. Dias S, Gama A, Simões D, Mendão L. Implementation process and impacts of a participatory HIV research project with key populations. Biomed Res Int 2018;2018:5845218 [FREE Full text] [CrossRef] [Medline]
  92. Fonseka TM, Pong JT, Kcomt A, Kennedy SH, Parikh SV. Collaborating with individuals with lived experience to adapt CANMAT clinical depression guidelines into a patient treatment guide: the CHOICE-D co-design process. J Eval Clin Pract 2020 Aug;26(4):1259-1269. [CrossRef] [Medline]
  93. Frith L, Hepworth L, Lowers V, Joseph F, Davies E, Gabbay M. Role of public involvement in the Royal College of Physicians' Future Hospitals healthcare improvement programme: an evaluation. BMJ Open 2019 Sep 12;9(9):e027680. [CrossRef] [Medline]
  94. King C, Gillard S. Bringing together coproduction and community participatory research approaches: using first person reflective narrative to explore coproduction and community involvement in mental health research. Health Expect 2019 Aug;22(4):701-708 [FREE Full text] [CrossRef] [Medline]
  95. Leslie M, Khayatzadeh-Mahani A, MacKean G. Recruitment of caregivers into health services research: lessons from a user-centred design study. Res Involv Engagem 2019;5:17 [FREE Full text] [CrossRef] [Medline]
  96. McCarron TL, Noseworthy T, Moffat K, Wilkinson G, Zelinsky S, White D, et al. Understanding the motivations of patients: a co-designed project to understand the factors behind patient engagement. Health Expect 2019 Aug;22(4):709-720 [FREE Full text] [CrossRef] [Medline]
  97. Taylor RM, Mohain J, Gibson F, Solanki A, Whelan J, Fern LA. Novel participatory methods of involving patients in research: naming and branding a longitudinal cohort study, BRIGHTLIGHT. BMC Med Res Methodol 2015 Mar 14;15:20 [FREE Full text] [CrossRef] [Medline]
  98. Woods L, Roehrer E, Duff J, Walker K, Cummings E. Co-design of a mobile health app for heart failure: perspectives from the team. Stud Health Technol Inform 2019 Aug 08;266:183-188. [CrossRef] [Medline]
  99. Tsianakas V, Maben J, Wiseman T, Robert G, Richardson A, Madden P, et al. Using patients' experiences to identify priorities for quality improvement in breast cancer care: patient narratives, surveys or both? BMC Health Serv Res 2012 Aug 22;12(1):271 [FREE Full text] [CrossRef] [Medline]
  100. Cheraghi-Sohi S, Hole AR, Mead N, McDonald R, Whalley D, Bower P, et al. What patients want from primary care consultations: a discrete choice experiment to identify patients' priorities. Ann Fam Med 2008;6(2):107-115 [FREE Full text] [CrossRef] [Medline]
  101. Kalot MA, Al-Khatib M, Connell NT, Flood V, Brignardello-Petersen R, James P, VWD working group. An international survey to inform priorities for new guidelines on von Willebrand disease. Haemophilia 2020 Jan 26;26(1):106-116 [FREE Full text] [CrossRef] [Medline]
  102. Brooks H, Irmansyah I, Susanti H, Utomo B, Prawira B, Iskandar L, et al. Evaluating the acceptability of a co-produced and co-delivered mental health public engagement festival: Mental Health Matters, Jakarta, Indonesia. Res Involv Engagem 2019;5:25 [FREE Full text] [CrossRef] [Medline]
  103. Jilka S, Murray C, Wieczorek A, Griffiths H, Wykes T, McShane R. Exploring patients' and carers' views about the clinical use of ketamine to inform policy and practical decisions: mixed-methods study. BJPsych Open 2019 Jul 30;5(5):e62 [FREE Full text] [CrossRef] [Medline]
  104. Troya MI, Dikomitis L, Babatunde OO, Bartlam B, Chew-Graham CA. Understanding self-harm in older adults: a qualitative study. EClinicalMedicine 2019 Jul;12:52-61 [FREE Full text] [CrossRef] [Medline]
  105. Bayliss K, Starling B, Raza K, Johansson EC, Zabalan C, Moore S, et al. Patient involvement in a qualitative meta-synthesis: lessons learnt. Res Involv Engagem 2016 May 12;2(1):18 [FREE Full text] [CrossRef] [Medline]
  106. Best P, Badham J, Corepal R, O'Neill RF, Tully MA, Kee F, et al. Network methods to support user involvement in qualitative data analyses: an introduction to Participatory Theme Elicitation. Trials 2017 Nov 23;18(1):559 [FREE Full text] [CrossRef] [Medline]
  107. Garfield S, Jheeta S, Husson F, Jacklin A, Bischler A, Norton C, et al. Lay involvement in the analysis of qualitative data in health services research: a descriptive study. Res Involv Engagem 2016;2:29 [FREE Full text] [CrossRef] [Medline]
  108. Jennings H, Slade M, Bates P, Munday E, Toney R. Best practice framework for Patient and Public Involvement (PPI) in collaborative data analysis of qualitative mental health research: methodology development and refinement. BMC Psychiatry 2018 Jun 28;18(1):213 [FREE Full text] [CrossRef] [Medline]
  109. Stevenson M, Taylor BJ. Involving individuals with dementia as co-researchers in analysis of findings from a qualitative study. Dementia (London) 2019 Feb;18(2):701-712. [CrossRef] [Medline]
  110. Thomas F, Hansford L, Wyatt K, Byng R, Coombes K, Finch J, et al. An engaged approach to exploring issues around poverty and mental health: a reflective evaluation of the research process from researchers and community partners involved in the DeStress study. Health Expect 2020 May 24. [CrossRef] [Medline]
  111. Lee C, Tanna N, Blair M, Yusuf Y, Khalief H, Lakhanpaul M. Getting underneath the skin: a community engagement event for optimal vitamin D status in an 'easily overlooked' group. Health Expect 2019 Dec;22(6):1322-1330 [FREE Full text] [CrossRef] [Medline]
  112. Kuluski K, Ho JW, Cadel L, Shearkhani S, Levy C, Marcinow M, et al. An alternate level of care plan: co-designing components of an intervention with patients, caregivers and providers to address delayed hospital discharge challenges. Health Expect 2020 Oct;23(5):1155-1165 [FREE Full text] [CrossRef] [Medline]
  113. Maccarthy J, Guerin S, Wilson AG, Dorris ER. Facilitating public and patient involvement in basic and preclinical health research. PLoS One 2019;14(5):e0216600 [FREE Full text] [CrossRef] [Medline]
  114. Garfield S, Jheeta S, Jacklin A, Bischler A, Norton C, Franklin BD. Patient and public involvement in data collection for health services research: a descriptive study. Res Involv Engagem 2015;1:8 [FREE Full text] [CrossRef] [Medline]
  115. Hoddinott P, Pollock A, O'Cathain A, Boyer I, Taylor J, MacDonald C, et al. How to incorporate patient and public perspectives into the design and conduct of research. F1000Res 2018;7:752 [FREE Full text] [CrossRef] [Medline]
  116. Morris M, Alencar Y, Rachet B, Stephens R, Coleman MP. Fleshing out the data: when epidemiological researchers engage with patients and carers. Learning lessons from a patient involvement activity. BMJ Open 2020 Sep 30;10(9):e036311 [FREE Full text] [CrossRef] [Medline]
  117. Chouvarda I, Maramis C, Livitckaia K, Trajkovik V, Burmaoglu S, Belani H, ENJECT Working Group 1 Network. Connected health services: framework for an impact assessment. J Med Internet Res 2019 Sep 03;21(9):e14005 [FREE Full text] [CrossRef] [Medline]
  118. Chapman A, Hadfield M, Chapman C. Qualitative research in healthcare: an introduction to grounded theory using thematic analysis. J R Coll Physicians Edinb 2015;45(3):201-205. [CrossRef]
  119. Ratzki-Leewing A, Parvaresh Rizi E, Harris SB. Family members: the forgotten players in the Diabetes Care Team (The TALK-HYPO Study). Diabetes Ther 2019 Dec;10(6):2305-2311 [FREE Full text] [CrossRef] [Medline]
  120. Carlin CS, Christianson JB, Keenan P, Finch M. Chronic illness and patient satisfaction. Health Serv Res 2012 Dec;47(6):2250-2272 [FREE Full text] [CrossRef] [Medline]
  121. Tran V, Messou E, Djima MM, Ravaud P, Ekouevi DK. Patients' perspectives on how to decrease the burden of treatment: a qualitative study of HIV care in sub-Saharan Africa. BMJ Qual Saf 2019 Apr;28(4):266-275 [FREE Full text] [CrossRef] [Medline]
  122. Morgan DL. Focus Groups as Qualitative Research. Thousand Oaks, CA: SAGE Publications Ltd; 1997.
  123. Longhurst R. Semi-structured interviews and focus groups. Key Methods in Geography 2003;3(2):143-156 [FREE Full text]
  124. Wiig S, Storm M, Aase K, Gjestsen MT, Solheim M, Harthug S, QUASER team. Investigating the use of patient involvement and patient experience in quality improvement in Norway: rhetoric or reality? BMC Health Serv Res 2013 Jun 06;13(1):206 [FREE Full text] [CrossRef] [Medline]
  125. Newcomer K, Hatry H, Wholey J. Conducting semi-structured interviews. In: Handbook of Practical Program Evaluation. Hoboken, New Jersey, United States: John Wiley & Sons; 2015.
  126. Cowan K. The James Lind Alliance Guidebook. National Institute for Health Research. 2018.   URL: [accessed 2021-12-07]
  127. Patient and Public Involvement (PPI) in a digital age : an overview of AHSN activity and learning. Academic Health Science Networks. 2018.   URL: https:/​/www.​​app/​uploads/​2018/​12/​Patient-and-public-involvement-PPI-in-a-digital-world-May-2018.​pdf [accessed 2021-12-07]
  128. Créquit P, Mansouri G, Benchoufi M, Vivot A, Ravaud P. Mapping of crowdsourcing in health: systematic review. J Med Internet Res 2018 May 15;20(5):e187 [FREE Full text] [CrossRef] [Medline]
  129. Truitt AR, Monsell SE, Avins AL, Nerenz DR, Lawrence SO, Bauer Z, et al. Prioritizing research topics: a comparison of crowdsourcing and patient registry. Qual Life Res 2018 Jan 5;27(1):41-50. [CrossRef] [Medline]
  130. Cameron D, Smith GA, Daniulaityte R, Sheth AP, Dave D, Chen L, et al. PREDOSE: a semantic web platform for drug abuse epidemiology using social media. J Biomed Inform 2013 Dec;46(6):985-997 [FREE Full text] [CrossRef] [Medline]
  131. Ampofo L, Collister S, O'Loughlin B, Chadwick A. Text mining and social media: when quantitative meets qualitative and software meets people. In: Halfpenny P, Procter R, editors. Innovations in Digital Research Methods. Thousand Oaks, CA: SAGE Publications Ltd; 2015:161-192.
  132. Design together, live better. The AHSN Network.   URL: [accessed 2020-11-20]
  133. Melton H, Meader N, Dale H, Wright K, Jones-Diette J, Temple M, et al. Interventions for adults with a history of complex traumatic events: the INCiTE mixed-methods systematic review. Health Technol Assess 2020 Sep;24(43):1-312 [FREE Full text] [CrossRef] [Medline]
  134. Brooks H, Irmansyah I, Lovell K, Savitri I, Utomo B, Prawira B, et al. Improving mental health literacy among young people aged 11-15 years in Java, Indonesia: co-development and feasibility testing of a culturally-appropriate, user-centred resource (IMPeTUs) - a study protocol. BMC Health Serv Res 2019 Jul 12;19(1):484 [FREE Full text] [CrossRef] [Medline]
  135. Lavallee DC, Lee JR, Semple JL, Lober WB, Evans HL. Engaging patients in co-design of mobile health tools for surgical site infection surveillance: implications for research and implementation. Surg Infect (Larchmt) 2019 Oct;20(7):535-540 [FREE Full text] [CrossRef] [Medline]
  136. Frisch N, Atherton P, Doyle-Waters MM, MacLeod ML, Mallidou A, Sheane V, et al. Patient-oriented research competencies in health (PORCH) for researchers, patients, healthcare providers, and decision-makers: results of a scoping review. Res Involv Engagem 2020;6:4 [FREE Full text] [CrossRef] [Medline]
  137. Price A, Vasanthan L, Clarke M, Liew SM, Brice A, Burls A. SMOOTH: Self-Management of Open Online Trials in Health analysis found improvements were needed for reporting methods of internet-based trials. J Clin Epidemiol 2019 Jan;105:27-39. [CrossRef] [Medline]
  138. Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. BMC Health Serv Res 2014 Feb 26;14(1):89 [FREE Full text] [CrossRef] [Medline]
  139. Ludwig C, Graham ID, Gifford W, Lavoie J, Stacey D. Partnering with frail or seriously ill patients in research: a systematic review. Res Involv Engagem 2020;6:52 [FREE Full text] [CrossRef] [Medline]
  140. Lucas PJ, Allnock D, Jessiman T. How are European birth-cohort studies engaging and consulting with young cohort members? BMC Med Res Methodol 2013 Apr 11;13:56 [FREE Full text] [CrossRef] [Medline]
  141. Mathie E, Wythe H, Munday D, Millac P, Rhodes G, Roberts N, et al. Reciprocal relationships and the importance of feedback in patient and public involvement: a mixed methods study. Health Expect 2018 Oct;21(5):899-908 [FREE Full text] [CrossRef] [Medline]
  142. Chambers E, Gardiner C, Thompson J, Seymour J. Patient and carer involvement in palliative care research: an integrative qualitative evidence synthesis review. Palliat Med 2019 Sep;33(8):969-984 [FREE Full text] [CrossRef] [Medline]
  143. O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak 2016 Sep 15;16(1):120 [FREE Full text] [CrossRef] [Medline]
  144. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016;4(2):e72 [FREE Full text] [CrossRef] [Medline]
  145. Liabo K, Boddy K, Bortoli S, Irvine J, Boult H, Fredlund M, et al. Public involvement in health research: what does 'good' look like in practice? Res Involv Engagem 2020;6:11 [FREE Full text] [CrossRef] [Medline]
  146. Nunn JS, Tiller J, Fransquet P, Lacaze P. Public involvement in global genomics research: a scoping review. Front Public Health 2019;7:79 [FREE Full text] [CrossRef] [Medline]
  147. Miah J, Dawes P, Edwards S, Leroi I, Starling B, Parsons S. Patient and public involvement in dementia research in the European Union: a scoping review. BMC Geriatr 2019 Aug 14;19(1):220 [FREE Full text] [CrossRef] [Medline]
  148. Hutchison K, Rogers W, Entwistle VA. Addressing deficits and injustices: the potential epistemic contributions of patients to research. Health Care Anal 2017 Dec 8;25(4):386-403. [CrossRef] [Medline]
  149. Buck D, Gamble C, Dudley L, Preston J, Hanley B, Williamson PR, EPIC Patient Advisory Group. From plans to actions in patient and public involvement: qualitative study of documented plans and the accounts of researchers and patients sampled from a cohort of clinical trials. BMJ Open 2014 Dec 04;4(12):e006400 [FREE Full text] [CrossRef] [Medline]
  150. Naidoo N, Nguyen VT, Ravaud P, Young B, Amiel P, Schanté D, et al. The research burden of randomized controlled trial participation: a systematic thematic synthesis of qualitative evidence. BMC Med 2020 Jan 20;18(1):6 [FREE Full text] [CrossRef] [Medline]
  151. Bereczky T. Patient advocacy -with a feeling patient citizenship -the description of an affective model of patient advocacy. Doctoral Dissertation, ELTE University, Hungary. 2019.   URL: https:/​/www.​​42773922/​Patient_Advocacy_With_a_Feeling_Patient_citizenship_The_description_of_an_affective_model_of_patient_advocacy [accessed 2021-12-07]
  152. Sacristan JA, Aguaron A, Avendaño C, Garrido P, Carrion J, Gutierrez A, et al. Patient involvement in clinical research: why, when, and how. Patient Prefer Adher 2016 Apr:631. [CrossRef]
  153. Huckvale K, Venkatesh S, Christensen H. Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety. NPJ Digit Med 2019;2:88 [FREE Full text] [CrossRef] [Medline]
  154. Algeo N, Hunter D, Cahill A, Dickson C, Adams J. Usability of a digital self-management website for people with osteoarthritis: a UK patient and public involvement study. Int J Ther Rehabil 2017 Feb 02;24(2):78-82. [CrossRef]

GRIPP2: revised version of Guidance for Reporting Involvement of Patients and the Public
PPI: patient and public involvement

Edited by R Kukafka; submitted 13.11.20; peer-reviewed by Z Aghaei, A Khaleghi; comments to author 17.12.20; revised version received 16.01.21; accepted 08.10.21; published 23.12.21


©Gloria A Aguayo, Catherine Goetzinger, Renza Scibilia, Aurélie Fischer, Till Seuring, Viet-Thi Tran, Philippe Ravaud, Tamás Bereczky, Laetitia Huiart, Guy Fagherazzi. Originally published in the Journal of Medical Internet Research (, 23.12.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.