This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Patient participation is essential in developing high-quality guidelines but faces practical challenges. Evidence on timing, methods, evaluations, and outcomes of methodologies for patient participation in guideline development is lacking.
To assess the feasibility of a wiki as a participatory tool for patients in the development of a guideline on infertility determined by (1) use of the wiki (number of page views and visitors), (2) benefits of the wiki (ie, number, content, and eligibility of the recommendations to be integrated into the guideline), and (3) patients’ facilitators of and barriers to adoption, and the potential challenges to be overcome in improving this wiki.
To obtain initial content for the wiki, we conducted in-depth interviews (n = 12) with infertile patients. Transcripts from the interviews were translated into 90 draft recommendations. These were presented on a wiki. Over 7 months, infertile patients were invited through advertisements or mailings to formulate new or modify existing recommendations. After modifying the recommendations, we asked patients to select their top 5 or top 3 recommendations for each of 5 sections on fertility care. Finally, the guideline development group assessed the eligibility of the final set of recommendations within the scope of the guideline. We used a multimethod evaluation strategy to assess the feasibility of the wiki as a participatory tool for patients in guideline development.
The wiki attracted 298 unique visitors, yielding 289 recommendations. We assessed the 21 recommendations ranked as the top 5 or top 3 for their eligibility for being integrated into the clinical practice guideline. The evaluation identified some challenges needed to be met to improve the wiki tool, concerning its ease of use, website content and layout, and characteristics of the wiki tool.
The wiki is a promising and feasible participatory tool for patients in guideline development. A modified version of this tool including new modalities (eg, automatically limiting the number and length of recommendations, using a fixed format for recommendations, including a motivation page, and adding a continuous prioritization system) should be developed and evaluated in a patient-centered design.
Having patients participate in clinical practice guideline (CPG) development is essential but challenging [
Several practical limitations could explain why patient participation is not common practice in CPG development. First, various methods for patient participation in CPG development can be used, such as conducting in-depth interviews or focus group meetings to explore patients’ preferences, asking patients’ representatives to comment on drafts of the CPG, or including patients’ representatives or patients in the CPG development group [
Web 2.0 tools offer opportunities to let nonorganized groups participate in a complex process such as CPG development [
In the Netherlands, fertility care is mostly publically arranged and provided by various professionals. First, fertility care is provided by the general practitioner and may be part of an initial fertility assessment after a prolonged time of unwanted nonconception. Second, the general practitioner can refer couples to a gynecologist in a general (secondary care) or a university (tertiary care) hospital to complete the fertility assessment, determine a cause of infertility, and define a suitable treatment policy. Third, if a severe male factor is diagnosed, a urologist may be consulted. Furthermore, since infertility has a high emotional and psychological impact, which also interferes with work, a psychologist and occupational physician are regularly engaged in the care pathway. In vitro fertilization and intracytoplasmic sperm injection are provided by 13 licensed hospitals (8 university hospitals, 4 general hospitals, and 1 private clinic). Ovulation induction and intrauterine insemination are performed in all Dutch hospitals. Ovulation induction, intrauterine insemination cycles, and the first three in vitro fertilization or intracytoplasmic sperm injection treatment cycles are reimbursed as part of the basic health care package according to the Health Insurance Act.
In February 2008, a collaboration of stakeholders (a general practitioner, 2 gynecologists, a urologist/sexologist, a clinical embryologist, a clinical chemical specialist, a medical psychologist, an occupational physician, 2 patients’ representatives, and a researcher) was set up to develop a national multidisciplinary paper-based CPG on infertility. CPGs are defined as sets of evidence- or consensus-based recommendations describing optimal patient care to assist health care professionals and patients in clinical decision making [
The objective of this study was to assess the feasibility of the wiki as a participatory tool for patient participation in CPG development. The feasibility of the wiki was determined by three end points: (1) use of the wiki and users’ characteristics (number of page views and visitors), (2) wiki content quality, particularly the assessment of various aspects of the final set of unique recommendations (ie, number, content, and their eligibility for integration into the CPG) for high-quality fertility care, and (3) wiki system quality (ie, patients’ facilitators of and barriers to adoption of this wiki as a participatory tool for direct patient involvement in CPG development, as well as potential suggestions for improvement).
We developed a conventional wiki website using MediaWiki software and made accessible through the Freya website, called FreyaWIKI [
To obtain the initial content for this wiki, we first conducted 12 semistructured in-depth interviews with infertile couples during different phases of care, from the first visit to the general practitioner, to (non)pregnant status after completing medically assisted reproduction techniques [
Division of the draft recommendations into sections and subsections determined the structure of the wiki. Draft recommendations were divided into 4 sections (EB,JK), consisting of 3 sections referring to the care delivered by the 3 most involved professionals and a general section for recommendations important to all professionals: general care, care delivered by a general practitioner, gynecological care, and urological care. To provide more structure in the wiki, the draft recommendations in each of these 4 sections were subdivided into 8 subsections (EB, JK) based on aspects of care that are known to be important to infertile patients: 3 medical-technical aspects (ie, examination, therapy, and referral), 4 patient-centered aspects (ie, organization of care, information provision, communication, and staff attitudes), and 1 general aspect (ie, general) [
We recruited participants for the wiki evaluation through mailings to members of Freya, the Dutch patients’ association for infertility; advertisements in Freya’s quarterly journal; links on websites of Freya and the professional societies (eg, general practitioners, gynecologists, urologists, and clinical embryologists); and links in social media (eg, Hyves, Twitter, and Facebook). In addition, we sent advertising posters to all 103 clinics offering fertility treatments in the Netherlands for their waiting rooms.
From May to December 2008, we presented the draft recommendations for fertility care on the wiki. Patients were invited to modify or refine these recommendations and to add new recommendations. During this process, we asked patients to subscribe voluntarily through an email address and to provide background characteristics for study purposes. After 2 months, when the number of recommendations started to increase, patients and patients’ representatives requested us to add 2 sections to the existing structure of the wiki: 1 regarding the care delivered by the laboratory (eg, recommendations regarding semen analysis), and 1 regarding the care delivered by the remaining professionals who were not represented in a separate section (eg, recommendations regarding the medical psychologist). Hence, we added 2 sections to the wiki: laboratory and remaining. Next, we recategorized recommendations from the general section regarding care delivered by the laboratory or care delivered by professionals other than the general practitioner, gynecologist or urologist. After this restructuring of the wiki, the general section contained only recommendations on fertility care in general, thus not referring to the care delivered by 1 of the professionals involved.
After 7 months, we modified the recommendations in several steps. First, we removed duplicate recommendations. Then, if necessary, we moved recommendations into the appropriate sections (EB, JK). Since all recommendations in the remaining section turned out to be more suited to other sections, we eliminated this section. Next, 2 researchers (EB, WN) and the chief executive of Freya (JK) independently assessed the implementability of all recommendations using the Guideline Implementability Appraisal (GLIA) instrument [
All patients visiting the wiki website were invited to prioritize their top 5 (modified) recommendations in each section (for the laboratory section, we asked them to identify their top 3 due to the small number of recommendations). This prioritization was privately conducted by assigning 5, 4, 3, 2, and 1 points for the most important recommendations for determining high-quality fertility care for each of the 5 sections and independently from the subsections.
Initially, the CPG development group had intended to integrate this final top selection of patients’ recommendations directly into the CPG. However, before integrating these recommendations, the entire CPG development group (n = 11) assessed the eligibility of the recommendations for inclusion in terms of the scope of the guideline.
To evaluate the feasibility of the wiki, we performed a multimethod evaluation study including three components [
Data on actual use of the wiki (eg, number of unique visitors, page views) were generated through log files on the website of the patient association (Freya). Unique visitors were determined by IP address logged and stored on the website.
To evaluate the content quality of the wiki, we assessed various aspects of the final set of unique recommendations, particularly the number of recommendations, their content, and their eligibility for integration into the CPG for high-quality fertility care.
To evaluate the quality of the wiki system and to identify facilitators, barriers, and potential areas of improvement, we conducted an online questionnaire. To gain insight into the thoughts underlying the resulting factors that formed potential facilitators of or barriers to adoption of the wiki and aspects of improvement, we conducted in-depth interviews with wiki users who completed the evaluation questionnaire.
During the prioritization phase, patients visiting the wiki website were invited to complete an online evaluation questionnaire. This questionnaire included items regarding users’ background characteristics (eg, age, type of infertility), use of the wiki (eg, number of visits), and factors that could potentially influence adoption of the wiki (quality of the wiki website, satisfaction, and net benefits) based on the relevant evaluation factors derived from the Human, Organization, and Technology-fit framework [
We first summarized the identified influencing factors on adoption of the wiki website and suggested potential areas of improvement. Next, we translated these into a topic list to guide the in-depth interviews. To get both confirmation of and saturation in the thoughts underlying the facilitators of and barriers to adoption and potential areas of improvement of the wiki, 1 researcher (EB) conducted semistructured in-depth interviews with wiki users by telephone. Participants in the questionnaire survey who left their email address were randomly recruited by email. The first part of the interview consisted of open-ended questions, related to thoughts underlying the identified influencing factors on adoption and potentials for improvement of the wiki. Next, patients were asked for additional influencing factors and suggestions for improving the wiki. Recruitment continued until saturation of data was achieved. Regarding the starting and stopping criteria according to Francis and colleagues [
We used SPSS 16.0 for Windows, Data Entry 4.0 (IBM Corporation, Somers, NY, USA) to perform descriptive statistical tests on the background characteristics of the wiki participants and to analyze patients’ top rankings of the recommendations. The final top selection of recommendations in each section was determined by identifying those with the highest sumscores derived. For analyzing the results of the online evaluation questionnaire, we grouped the responses on the 5-point Likert scale into the categories agree (scores 1and 2), neutral (score 3), and disagree (scores 4 and 5). Items were a priori identified as facilitators of adoption if >50% chose agree (scores 1 and 2) and as barriers to adoption of the wiki website if >50% of the evaluators chose disagree (scores 4 and 5). We used the reported top three advantages and disadvantages and the potential areas of improvement of the wiki to determine the frequency of occurrence of each aspect. We conducted an initial content analysis of all free-text responses to the questionnaire, to determine additional points to be improved (EB, WN).
All interviews were audiotaped and transcribed verbatim. Data were analyzed iteratively and thematically across accounts (EB, JK) [
From the transcripts of 12 in-depth interviews with infertile patients, we translated the perceived bottlenecks into a set of 90 draft patient recommendations and entered them into the wiki (
For example, patients perceived a bottleneck in that appointments were possible only during working hours instead of also during the evening, which resulted in difficulties with work. The resulting draft recommendation was formulated as “I want the hospital to provide possibilities to make appointments during evening hours.” Other examples of the bottlenecks mentioned were the variation between hospitals’ laboratories in performing a semen analysis, unavailability of separate waiting rooms for pregnancy and infertility consultations, and gynecologists’ lack of empathy.
The process of obtaining recommendations for clinical practice guideline (CPG) development.
FreyaWIKI was structured through the division of recommendations into 6 sections. Each of these sections was subdivided into 8 subsections (
During 7 months of access, 36,473 wiki pages were viewed. We identified 298 unique users, including 81 registered users who provided background characteristics (
Screenshot of recommendations on FreyaWIKI.
Screenshot of the FreyaWIKI homepage.
Flow of wiki participants through the study.
Overall, we collected 265 recommendations and modified them into 289 unique recommendations (
Of the 80 patients who participated in the prioritization, 45 completed the questionnaire. Of these, 53% (n = 24) visited FreyaWIKI for the first time while completing the questionnaire, and 93% (n = 42) had never worked with a wiki, other than this one, before. Other background characteristics of the respondents are presented in
Reported advantages of the wiki were the privacy they experienced on the website, the structure of the website linking recommendations to sections on care delivered by fertility professionals, ease of navigation through the website, and the additional value of the wiki website as a source of information and as an opportunity to provide feedback to the care services.
Reported disadvantages of the wiki concerned the content of the wiki website, in terms of the unstructured recommendations not being formulated in a similar way, too much content being visible on one screen, and the nonattractive layout of the wiki website.
The main potential areas of improvement were providing information on treatment options and causal factors of infertility on the wiki website, broadening the marketing of the wiki by placing advertisements in commercial magazines, and communicating information on related activities (
Overall, 11 of the 30 patients who gave their email address in the evaluation questionnaire agreed to participate in the interviews. We conducted 3 interviews. All 3 interviews confirmed barriers to adoption as well as suggestions to improve the wiki, and saturation of the related underlying thoughts was reached (
Final set of the patients’ top-5 and top-3 recommendations (n = 23) for the 5 sections, ranked by importance to the quality of fertility care as formulated on the wiki website.
Section, rank, and recommendation | Subsection |
||
|
|||
1 | I want insurance companies to reimburse six attempts at in vitro fertilizationb | General | |
2 | I want insurance companies to start counting in vitro fertilization attempts only after oocyte retrieval or even after embryo transfer has been performedb | General | |
3 | I want my doctor to practice empathy, instead of only working on the technical or financial part | Staff attitudes | |
4 | I want the hospital to have separate waiting rooms for pregnant women and couples being treated for infertility | Organization | |
5 | I want to be able to arrange appointments during the daytime as well as in the evenings | Organization | |
|
|||
1 | I want my gynecologist and GPc to have good communication, so my GP knows what is going on with us | Referral | |
2 | I want my GP to make a referral immediately after we have been trying to conceive for a year | Referral | |
3 | I want to have my first medical consultation with my gynecologist within 1 month after referral. | Organization | |
4 | I want my GP to be informed of possible causes of infertility, in both women and men | General | |
5 | I want my GP to pay attention to nonmedical issues, such as stress, anxiety, relational problems, and sexuality | Communication | |
|
|||
1 | I want also to be able to receive treatments on weekends (Saturdays and Sundays) | Organization | |
2 | I want all members of the fertility treatment team to apply one policy regarding my infertility treatment | Organization | |
3 | I want my gynecologist to inform me of all possible fertility treatment options, even if these are outside the hospital | Information provision | |
4 | I want my gynecologist to inform me about the different phases of treatment and their expected time span | Information provision | |
5 | I want assisted hatching to be possible or available in the Netherlands | Therapy | |
|
|||
1 | I want my urologist and gynecologist to have good communication | Organization | |
2 | I want to be informed of the investigations that are to be performed by the urologist | Examination | |
3 | I want to have a permanent urologist who is specialized in infertility | Organization | |
4 | I want to have a consultation with a urologist within 1 month after referral | Organization | |
5 | I want my urologist to involve my partner in the conversation | Communication | |
|
|||
1 | I want to be informed as soon as possible when our embryos do not divide correctly | Organization | |
2 | I want Dutch laboratories to share protocols and learn from each other’s experiences | Organization | |
3 | I want to be informed of the causes of nonviability of our frozen embryos, if appropriate | Organization |
a Subsections were derived from the website’s structure and defined by the user.
b Recommendation was excluded, since it fell out of the scope of the clinical practice guideline.
c General practitioner.
Background characteristics of respondents (n = 45) to the evaluation questionnaire.
Characteristic | Data | |
|
||
Male | 0 (0%) | |
Female | 45 (100%) | |
Age (years), mean (SD) | 35 (5.24) | |
|
||
Primary | 15 (33%) | |
Secondary | 30 (67%) | |
Duration of infertility (months), median (range) | 36 (0–71) | |
|
||
Gynecologic | 19 (42%) | |
No pregnancy after fertility treatment | 8 (18%) | |
Pregnant achieved by fertility treatment | 4 (9%) | |
Unknown | 14 (31%) | |
|
||
Low | 0 (0%) | |
Intermediate | 14 (31%) | |
High | 31 (69%) | |
Membership in Freya, n (%) | 24 (53%) |
Patients’ barriers to adoption of the wiki (n = 45).
Factor influencing adoption of the wiki | Proportion disagreeing |
||
n | % | ||
|
|||
Findability of the website | 37 | 82% | |
Accessibility of the website | 35 | 78% | |
Clarity of log-in location on the website | 27 | 60% | |
Clarity on the goal of the website | 28 | 62% | |
Clarity on instructions for using the website | 24 | 53% | |
Efficiency of the website (ie, speed at which the website enabled users to accurately and successfully add and modify recommendations) | 24 | 53% | |
|
|||
Comprehensiveness of the clarifying text on the website | 30 | 66% | |
Satisfaction with the content of the formulated recommendations | 25 | 56% | |
Usefulness of clustering recommendations into sections in searching for existing recommendations | 23 | 51% | |
Similarity between formulated recommendations and participants’ actual opinions on fertility care | 23 | 51% | |
|
|||
Suitability of the wiki for obtaining recommendations for clinical practice guideline development | 32 | 71% | |
Ease of using the wiki | 24 | 53% | |
Accessibility of the wiki | 27 | 60% |
a Number (%) of participants who rated the positively formulated statements on the evaluation factors as disagree (scores 4 or 5).
Participants’ (n = 45a) suggestions for improving the wiki website.
Aspect of improvement | Respondents |
Sample quotes (translated from Dutch) from |
||
n | % | |||
|
||||
Findability of the website | 10 | 22% | Q: |
|
Accessibility of the website | 2 | 4% | I: |
|
|
||||
Comprehensiveness of clarifying text | 1 | 2% | Q: |
|
Clearness of description of the goal of the wiki | 4 | 8% | I: |
|
Clearness of instructions for use | 1 | 2% | ||
Satisfaction with formulated recommendations | 8 | 16% | I:... |
|
Similarity between actual preferences and recommendations | 4 | 8% | Q: |
|
Clarity of the structure in which recommendations are placed on the website | 30 | 66% | I: |
|
Relationship between length and number of recommendations and their presentation on one screen | 32 | 71% | Q: |
|
Education provision on the website | 19 | 42% | Q: |
|
|
||||
Usability of wiki methodology | 6 | 13% | Q: |
|
|
||||
Efficiency of wiki methodology | 5 | 11% | Q: |
|
|
||||
Impression of the layout | 33 | 73% | Q: |
|
|
||||
Marketing | 6 | 13% | Q: |
|
Community feeling of the wiki | 3 | 6% | I: |
a 45 participants completed the online evaluation questionnaire, of whom 3 participated in the in-depth interviews.
In this study, we showed that the wiki is a feasible tool to ensure active patient participation in the development of a Dutch multidisciplinary CPG on infertility. The high numbers of page views (36,473), unique visitors (298), and recommendations formulated (289) implies patients’ willingness and ability to contribute to CPGs through a wiki-based method. We also showed that such a wiki is a useful information source for patients.
Second, we gained a final set of 21 selected recommendations, which were assessed as being eligible to be integrated directly and transparently into the CPG. Third, patients had positive views on the experienced privacy, ease of navigation, divisional structure of the wiki, and its potential befits. A total of 98% of the patients would recommend the website and 84% would participate again in a similar project. This study also provided some important suggestions to improve this participatory tool for patients in the development of CPGs, concerning ease of use, content and layout of the website, and characteristics of the wiki tool.
Several studies on specialized medical wikis (eg, wikis that fall outside the scope of a general encyclopedia) have been published, but most particularly focus on education of medical students [
Professionals and patients’ representatives could also use the wiki and had the opportunity to informed themselves about patients’ views and to bring up content for discussion in the CPG development group. According to the results of the evaluation questionnaire and the interviews, this content was also helpful to patients as an information source, which may also explain the relatively large number of page views. Although providing information was not the initial goal of this wiki, its relevance is in agreement with published literature on conventional wikis [
Although drawbacks to active patient participation methods were reduced, this study drew attention to some other potential implications derived from patients’ suggestions that might improve the use of a future medical dedicated wiki exclusively for actively involving patients in CPG development. First, structuring recommendations and limiting the number and length of recommendations to presentation on one screen may improve usability [
This wiki has been tested in the field of infertility care, representing a relatively young target group [
Third, the participation rate in the evaluation of the wiki might have subjected our study to a participation bias of potentially the most motivated wiki users. However, this is a known limitation in the active use of wikis in general: the most motivated users provide most of the content [
The wiki is a promising and feasible tool to actively involve patients in CPG development. To improve the tool’s ease of use and practical aspects to enhance direct integration of recommendations into the CPG, a more specialized and refined wiki should be developed. This should include new modalities, such as automatically shortening the number and length of recommendations, using a fixed format for formulation of recommendations, using a continuous prioritization system for selection of the most important recommendations, and including a separate motivation page. Furthermore, in the development, attention should be paid to the informational character of such a wiki. To improve future implementability, a modified tool should preferably be codeveloped and evaluated by patients in a user-centered design study. Furthermore, representativeness of patients and recommendations should be integrated into this next phase.
Constructs of the online evaluation questionnaire.
Screenshot of recommendations on FreyaWIKI.
Screenshot of the FreyaWIKI homepage.
Appraisal of Guidelines Research and Evaluation
clinical practice guideline
Guideline Implementability Appraisal
The authors thank all participants of FreyaWIKI.
Funding: This project was funded by the Netherlands Organization for Health Research and Development (ZonMW), grant no. 150020015, The Hague. This open access publication was funded by the Dutch Organization for Scientific Research (NWO), grant no. 036.001.927. Both funders did not have any role in the design and conduct of the study, the collection, management, analysis, and interpretation of data, or in the preparation, review, or approval of the manuscript.
All researchers are independent from the funders.
Previous presentation/report: The content of this paper has not been copyrighted, published, or presented previously and is not under consideration for publication elsewhere.
Ethics approval: In this study, ethics approval was not required. However, all participants gave informed consent before taking part. All observational data and transcripts of interviews were anonymized.
EB was guarantor, designed the study, modified the recommendations, conducted and (qualitatively) analyzed the online evaluation questionnaire and in-depth interviews, interpreted the analysis of the use of the wiki and its evaluation, and drafted the paper. WN designed the study, checked the analysis, interpreted the analysis, and drafted the paper. JK designed the study, modified the recommendations, interpreted the analysis, and drafted the paper. JB designed the study, interpreted the analysis, and drafted the paper. RH designed the study, interpreted the analysis, and drafted the paper. JAMK led the research team, designed the study, was principal investigator, interpreted the analysis, and drafted the paper. All authors reviewed consecutive drafts of the paper and gave their final approval of the version to be published.
None declared.