Published on 26.10.12 in Vol 14, No 5 (2012): Sep-Oct
Feasibility of a Wiki as a Participatory Tool for Patients in Clinical Guideline Development
Background: Patient participation is essential in developing high-quality guidelines but faces practical challenges. Evidence on timing, methods, evaluations, and outcomes of methodologies for patient participation in guideline development is lacking.
Objective: To assess the feasibility of a wiki as a participatory tool for patients in the development of a guideline on infertility determined by (1) use of the wiki (number of page views and visitors), (2) benefits of the wiki (ie, number, content, and eligibility of the recommendations to be integrated into the guideline), and (3) patients’ facilitators of and barriers to adoption, and the potential challenges to be overcome in improving this wiki.
Methods: To obtain initial content for the wiki, we conducted in-depth interviews (n = 12) with infertile patients. Transcripts from the interviews were translated into 90 draft recommendations. These were presented on a wiki. Over 7 months, infertile patients were invited through advertisements or mailings to formulate new or modify existing recommendations. After modifying the recommendations, we asked patients to select their top 5 or top 3 recommendations for each of 5 sections on fertility care. Finally, the guideline development group assessed the eligibility of the final set of recommendations within the scope of the guideline. We used a multimethod evaluation strategy to assess the feasibility of the wiki as a participatory tool for patients in guideline development.
Results: The wiki attracted 298 unique visitors, yielding 289 recommendations. We assessed the 21 recommendations ranked as the top 5 or top 3 for their eligibility for being integrated into the clinical practice guideline. The evaluation identified some challenges needed to be met to improve the wiki tool, concerning its ease of use, website content and layout, and characteristics of the wiki tool.
Conclusions: The wiki is a promising and feasible participatory tool for patients in guideline development. A modified version of this tool including new modalities (eg, automatically limiting the number and length of recommendations, using a fixed format for recommendations, including a motivation page, and adding a continuous prioritization system) should be developed and evaluated in a patient-centered design.
J Med Internet Res 2012;14(5):e138
Having patients participate in clinical practice guideline (CPG) development is essential but challenging [, ]. Their participation is particularly assumed to result in higher-quality guidelines in terms of applicability, acceptability, usefulness, and enhancement of implementation [ - ]. For instance, patient participation is one of the key criteria of the Appraisal of Guidelines Research and Evaluation (AGREE) instrument [ ], which is used to assess the methodological quality of guidelines. However, only 25%-50% of CPG developers regularly involve patients [ ].
Several practical limitations could explain why patient participation is not common practice in CPG development. First, various methods for patient participation in CPG development can be used, such as conducting in-depth interviews or focus group meetings to explore patients’ preferences, asking patients’ representatives to comment on drafts of the CPG, or including patients’ representatives or patients in the CPG development group [, , - ]. However, practical guidance on how and when to apply these methods is lacking [ ]. Second, all methods are restricted to including a selected number of patients or patients’ representatives and do not involve a large population of patients. Third, transparently integrating patients’ preferences into CPG recommendations is difficult and often unclear [ ]. Fourth, organizational (eg, recruitment of participants), financial (eg, costs of patients’ education or for conducting focus groups), and sociopolitical barriers (eg, CPG developers’ resistance to including patients in the CPG group) may impede patient participation in CPG development [ ]. Finally, studies on the effectiveness and impact of patient participation are limited [ ]. A new methodology for patient participation in CPG development that enables overcoming most of these drawbacks is thus necessary.
Web 2.0 tools offer opportunities to let nonorganized groups participate in a complex process such as CPG development [- ]. In particular a wiki, such as Wikipedia, seems to be an easily accessed tool, which enables patients to collaborate in formulating guideline recommendations directly. Ideally, to test the feasibility of such a new method for patient participation in CPG development, an Internet-using young target group such as infertile patients [ - ] is preferred. Infertility is commonly defined as “any form of reduced fertility with prolonged time of unwanted non-conception” [ ] and affects approximately 80 million couples worldwide [ , ]. In this study, we applied wiki technology as a participatory tool for patients in the development of a multidisciplinary CPG on infertility and aimed to assess its feasibility.
In the Netherlands, fertility care is mostly publically arranged and provided by various professionals. First, fertility care is provided by the general practitioner and may be part of an initial fertility assessment after a prolonged time of unwanted nonconception. Second, the general practitioner can refer couples to a gynecologist in a general (secondary care) or a university (tertiary care) hospital to complete the fertility assessment, determine a cause of infertility, and define a suitable treatment policy. Third, if a severe male factor is diagnosed, a urologist may be consulted. Furthermore, since infertility has a high emotional and psychological impact, which also interferes with work, a psychologist and occupational physician are regularly engaged in the care pathway. In vitro fertilization and intracytoplasmic sperm injection are provided by 13 licensed hospitals (8 university hospitals, 4 general hospitals, and 1 private clinic). Ovulation induction and intrauterine insemination are performed in all Dutch hospitals. Ovulation induction, intrauterine insemination cycles, and the first three in vitro fertilization or intracytoplasmic sperm injection treatment cycles are reimbursed as part of the basic health care package according to the Health Insurance Act.
In February 2008, a collaboration of stakeholders (a general practitioner, 2 gynecologists, a urologist/sexologist, a clinical embryologist, a clinical chemical specialist, a medical psychologist, an occupational physician, 2 patients’ representatives, and a researcher) was set up to develop a national multidisciplinary paper-based CPG on infertility. CPGs are defined as sets of evidence- or consensus-based recommendations describing optimal patient care to assist health care professionals and patients in clinical decision making . The aim of the CPG was to focus on organizational and patient-centered aspects of fertility care. Two representatives of the Dutch patients’ association for infertility, Freya, participated in the CPG development group. However, for direct patient participation in this guideline, we applied a wiki concurrently with this guideline development phase.
The objective of this study was to assess the feasibility of the wiki as a participatory tool for patient participation in CPG development. The feasibility of the wiki was determined by three end points: (1) use of the wiki and users’ characteristics (number of page views and visitors), (2) wiki content quality, particularly the assessment of various aspects of the final set of unique recommendations (ie, number, content, and their eligibility for integration into the CPG) for high-quality fertility care, and (3) wiki system quality (ie, patients’ facilitators of and barriers to adoption of this wiki as a participatory tool for direct patient involvement in CPG development, as well as potential suggestions for improvement).
Wiki Tool Development
We developed a conventional wiki website using MediaWiki software and made accessible through the Freya website, called FreyaWIKI . During the preparation phase, we first conducted in-depth interviews to obtain initial content for the wiki. Next, we structured the wiki tool according to the topics of the recommendations derived from the interviews.
Obtaining Initial Content of the Wiki Tool From In-Depth Interviews
To obtain the initial content for this wiki, we first conducted 12 semistructured in-depth interviews with infertile couples during different phases of care, from the first visit to the general practitioner, to (non)pregnant status after completing medically assisted reproduction techniques . Patients visiting outpatient clinics in Nijmegen and Amsterdam were consecutively invited to participate through an information letter. Subsequently, 1 researcher (EB) obtained the final consent by telephone. Participants were asked to specify perceived bottlenecks in their fertility care pathway, using an interview guide including the main treatment stages of their fertility care pathway (eg, treatment by a general practitioner, gynecologist, or urologist). Interviews were audiotaped and transcribed verbatim. Next, a researcher (EB) and the chief executive of Freya (JK) independently translated these bottlenecks into draft patient recommendations. These draft recommendations were formulated as “I want my doctor to....” Consensus on the formulation of patient draft recommendations was reached through discussion.
Structuring the Wiki Tool
Division of the draft recommendations into sections and subsections determined the structure of the wiki. Draft recommendations were divided into 4 sections (EB,JK), consisting of 3 sections referring to the care delivered by the 3 most involved professionals and a general section for recommendations important to all professionals: general care, care delivered by a general practitioner, gynecological care, and urological care. To provide more structure in the wiki, the draft recommendations in each of these 4 sections were subdivided into 8 subsections (EB, JK) based on aspects of care that are known to be important to infertile patients: 3 medical-technical aspects (ie, examination, therapy, and referral), 4 patient-centered aspects (ie, organization of care, information provision, communication, and staff attitudes), and 1 general aspect (ie, general) . These subsections were presented on the wiki in the following order: general (recommendations in general and those that don’t apply to other care aspects), information provision (recommendations on oral and paper-based information provision), organization (recommendations on the organization of fertility care, for example, adjustment of care between different health care professionals, accessibility of care), staff attitudes (recommendations on the attitude of health care professionals toward the patient, for example, having empathy), communication (recommendations on communication between the health care professional and the patient), examination (recommendations on examinations during fertility care), therapy (recommendations on therapy, namely infertility treatment by, for example, in vitro fertilization), and referral (recommendations on referral from one health care professional to another, for example, from a general practitioner to a gynecologist). Discrepancies in division and subdivision of the draft recommendations were resolved through discussion.
Patient Participation in CPG Development
We recruited participants for the wiki evaluation through mailings to members of Freya, the Dutch patients’ association for infertility; advertisements in Freya’s quarterly journal; links on websites of Freya and the professional societies (eg, general practitioners, gynecologists, urologists, and clinical embryologists); and links in social media (eg, Hyves, Twitter, and Facebook). In addition, we sent advertising posters to all 103 clinics offering fertility treatments in the Netherlands for their waiting rooms.
Obtaining Recommendations From Wiki Participants
From May to December 2008, we presented the draft recommendations for fertility care on the wiki. Patients were invited to modify or refine these recommendations and to add new recommendations. During this process, we asked patients to subscribe voluntarily through an email address and to provide background characteristics for study purposes. After 2 months, when the number of recommendations started to increase, patients and patients’ representatives requested us to add 2 sections to the existing structure of the wiki: 1 regarding the care delivered by the laboratory (eg, recommendations regarding semen analysis), and 1 regarding the care delivered by the remaining professionals who were not represented in a separate section (eg, recommendations regarding the medical psychologist). Hence, we added 2 sections to the wiki: laboratory and remaining. Next, we recategorized recommendations from the general section regarding care delivered by the laboratory or care delivered by professionals other than the general practitioner, gynecologist or urologist. After this restructuring of the wiki, the general section contained only recommendations on fertility care in general, thus not referring to the care delivered by 1 of the professionals involved.
After 7 months, we modified the recommendations in several steps. First, we removed duplicate recommendations. Then, if necessary, we moved recommendations into the appropriate sections (EB, JK). Since all recommendations in the remaining section turned out to be more suited to other sections, we eliminated this section. Next, 2 researchers (EB, WN) and the chief executive of Freya (JK) independently assessed the implementability of all recommendations using the Guideline Implementability Appraisal (GLIA) instrument . Discrepancies were discussed and resolved through consensus. Based on the results of this assessment, the recommendations were independently textually refined or modified by a researcher (EB) and the chief executive of Freya (JK). Finally, after consensus was reached on the final formulation, we reentered the recommendations into the wiki.
All patients visiting the wiki website were invited to prioritize their top 5 (modified) recommendations in each section (for the laboratory section, we asked them to identify their top 3 due to the small number of recommendations). This prioritization was privately conducted by assigning 5, 4, 3, 2, and 1 points for the most important recommendations for determining high-quality fertility care for each of the 5 sections and independently from the subsections.
Assessing Eligibility of the Selected Recommendations
Initially, the CPG development group had intended to integrate this final top selection of patients’ recommendations directly into the CPG. However, before integrating these recommendations, the entire CPG development group (n = 11) assessed the eligibility of the recommendations for inclusion in terms of the scope of the guideline.
Evaluation of the Wiki
To evaluate the feasibility of the wiki, we performed a multimethod evaluation study including three components . First, to assess the ability to involve large and diverse patient populations compared with other methods such as interviews, we evaluated wiki use and users’ characteristics. Second, we evaluated wiki content quality (ie, recommendations) and, third, wiki system quality (eg, ease of use, layout), identifying factors that could potentially influence adoption of the wiki (barriers and facilitators) as well as potential factors for improvement.
Evaluation of Wiki Use and Users’ Characteristics
Data on actual use of the wiki (eg, number of unique visitors, page views) were generated through log files on the website of the patient association (Freya). Unique visitors were determined by IP address logged and stored on the website.
Evaluation of Wiki Content Quality
To evaluate the content quality of the wiki, we assessed various aspects of the final set of unique recommendations, particularly the number of recommendations, their content, and their eligibility for integration into the CPG for high-quality fertility care.
Evaluation of Wiki System Quality
To evaluate the quality of the wiki system and to identify facilitators, barriers, and potential areas of improvement, we conducted an online questionnaire. To gain insight into the thoughts underlying the resulting factors that formed potential facilitators of or barriers to adoption of the wiki and aspects of improvement, we conducted in-depth interviews with wiki users who completed the evaluation questionnaire.
Online Evaluation Questionnaire
During the prioritization phase, patients visiting the wiki website were invited to complete an online evaluation questionnaire. This questionnaire included items regarding users’ background characteristics (eg, age, type of infertility), use of the wiki (eg, number of visits), and factors that could potentially influence adoption of the wiki (quality of the wiki website, satisfaction, and net benefits) based on the relevant evaluation factors derived from the Human, Organization, and Technology-fit framework . Questions on the potential influencing factors were grouped into 5 sections: ease of use of the wiki website, layout of the wiki website, value of the wiki methodology as a participatory tool for CPG development, content of the wiki website, and experienced privacy on the wiki website. Patients were asked to rate 22 accompanying positively formulated statements on these factors on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree) ( ). After each section, patients were invited to comment. Next, patients were asked to describe their three advantages and disadvantages of the wiki website and potential areas of improvement. Finally, patients were asked for their willingness to participate again in a similar project and for their intention to recommend this wiki.
In-Depth Interviews With Wiki Users
We first summarized the identified influencing factors on adoption of the wiki website and suggested potential areas of improvement. Next, we translated these into a topic list to guide the in-depth interviews. To get both confirmation of and saturation in the thoughts underlying the facilitators of and barriers to adoption and potential areas of improvement of the wiki, 1 researcher (EB) conducted semistructured in-depth interviews with wiki users by telephone. Participants in the questionnaire survey who left their email address were randomly recruited by email. The first part of the interview consisted of open-ended questions, related to thoughts underlying the identified influencing factors on adoption and potentials for improvement of the wiki. Next, patients were asked for additional influencing factors and suggestions for improving the wiki. Recruitment continued until saturation of data was achieved. Regarding the starting and stopping criteria according to Francis and colleagues , we started with 2 interviews and aimed to repeat cycles of 2 interviews until we obtained no new data. If data saturation was achieved, an additional interview was conducted to attain data saturation.
We used SPSS 16.0 for Windows, Data Entry 4.0 (IBM Corporation, Somers, NY, USA) to perform descriptive statistical tests on the background characteristics of the wiki participants and to analyze patients’ top rankings of the recommendations. The final top selection of recommendations in each section was determined by identifying those with the highest sumscores derived. For analyzing the results of the online evaluation questionnaire, we grouped the responses on the 5-point Likert scale into the categories agree (scores 1and 2), neutral (score 3), and disagree (scores 4 and 5). Items were a priori identified as facilitators of adoption if >50% chose agree (scores 1 and 2) and as barriers to adoption of the wiki website if >50% of the evaluators chose disagree (scores 4 and 5). We used the reported top three advantages and disadvantages and the potential areas of improvement of the wiki to determine the frequency of occurrence of each aspect. We conducted an initial content analysis of all free-text responses to the questionnaire, to determine additional points to be improved (EB, WN).
Qualitative Analysis of the Interviews
All interviews were audiotaped and transcribed verbatim. Data were analyzed iteratively and thematically across accounts (EB, JK) , according to the relevant factors of the evaluation framework, as used in the questionnaire to identify barriers to and facilitators of adoption and potential areas of improvement of the wiki [ ]. Another researcher (WN) independently checked the coding framework and analysis.
Wiki Tool Development
From the transcripts of 12 in-depth interviews with infertile patients, we translated the perceived bottlenecks into a set of 90 draft patient recommendations and entered them into the wiki ().
For example, patients perceived a bottleneck in that appointments were possible only during working hours instead of also during the evening, which resulted in difficulties with work. The resulting draft recommendation was formulated as “I want the hospital to provide possibilities to make appointments during evening hours.” Other examples of the bottlenecks mentioned were the variation between hospitals’ laboratories in performing a semen analysis, unavailability of separate waiting rooms for pregnancy and infertility consultations, and gynecologists’ lack of empathy.
Structure of the Wiki
FreyaWIKI was structured through the division of recommendations into 6 sections. Each of these sections was subdivided into 8 subsections (and , see and for translations).
Patient Participation in CPG Development
Wiki Use and Users’ Characteristics
During 7 months of access, 36,473 wiki pages were viewed. We identified 298 unique users, including 81 registered users who provided background characteristics (). The majority of them were female 78/81 (96%), highly educated 54/81(67%), and middle aged (mean 33 years). Median duration of infertility was 30 months (range 0–71 months). More than half 43/81(53%) underwent medically assisted reproduction techniques during the period of their visit. Another 14% (n = 11) stayed childless despite treatment.
Wiki Content Quality
Overall, we collected 265 recommendations and modified them into 289 unique recommendations (). After patients (n = 80) prioritized the recommendations by ranking the top 5 or top 3 in each section (we had eliminated the remaining section) according to their importance to high-quality fertility care, we selected 23 recommendations (4 sets of top 5 and 1 set of top 3) for eligibility assessment by the CPG development group ( ). We excluded 2 insurance-related recommendations, since they did not meet the scope of the CPG. The CPG development group accepted all of the remaining 21 recommendations, which were directly integrated into the CPG. More than half (n = 11) of the final set of recommendations concerned the organization of care. Similar to the quality assessment scale used in evidence-based recommendations (levels of evidence A-D) [ ], a level of P (patients) was provided for the patients’ recommendations and formulated as “ Patients would like to….” Participants were informed by email, on the wiki website, and through the websites of Freya and the professional associations of the final CPG that included the untouched eligible recommendations of the patients.
Evaluation of Wiki System Quality
Online Evaluation Questionnaire
Of the 80 patients who participated in the prioritization, 45 completed the questionnaire. Of these, 53% (n = 24) visited FreyaWIKI for the first time while completing the questionnaire, and 93% (n = 42) had never worked with a wiki, other than this one, before. Other background characteristics of the respondents are presented in. Facilitators of adoption of the wiki, defined as >50% of respondents agreeing (scores 1 and 2) to the relevant statements, were not identified. Barriers, defined as >50% disagreeing (scores 4 and 5) to the relevant statements, were identified in 3 of the 5 sections: ease of use, content of the website, and value of the wiki methodology ( ). In decreasing order of the proportion of evaluators who disagreed with the relevant statement, the main identified barriers concerned the findability (82%) and accessibility (78%) of the website, and the suitability of this wiki for obtaining recommendations for CPG development (71%).
Reported advantages of the wiki were the privacy they experienced on the website, the structure of the website linking recommendations to sections on care delivered by fertility professionals, ease of navigation through the website, and the additional value of the wiki website as a source of information and as an opportunity to provide feedback to the care services.
Reported disadvantages of the wiki concerned the content of the wiki website, in terms of the unstructured recommendations not being formulated in a similar way, too much content being visible on one screen, and the nonattractive layout of the wiki website.
The main potential areas of improvement were providing information on treatment options and causal factors of infertility on the wiki website, broadening the marketing of the wiki by placing advertisements in commercial magazines, and communicating information on related activities (). Overall, 98% of the patients said they would recommend the website and 84% would participate again in a similar project.
Overall, 11 of the 30 patients who gave their email address in the evaluation questionnaire agreed to participate in the interviews. We conducted 3 interviews. All 3 interviews confirmed barriers to adoption as well as suggestions to improve the wiki, and saturation of the related underlying thoughts was reached (). All patients reported problems with formulating a recommendation and expressed their wish to add a personal touch to the recommendation (eg, to explain why something should be done). The introduction of a motivation page, where patients could describe why they formulated a recommendation, might meet this request. Patients also embraced the missing community feeling as mentioned in the evaluation questionnaire. Introducing a monthly newsletter and automatically sending an email to the person who made the recommendation were suggested. All 3 interviewees regarded the website as a valuable source of information, rather than as a tool for modifying recommendations for CPG development. They mentioned that the content of the wiki had been helpful to them in searching for information on experiences regarding infertility treatment and in searching for recognition of their own experiences. Challenges faced by users in understanding the purpose of the website would be addressed by clearer instructions.
|Section, rank, and recommendation|| Subsection |
(aspect of care)a
|1||I want insurance companies to reimburse six attempts at in vitro fertilizationb||General|
|2||I want insurance companies to start counting in vitro fertilization attempts only after oocyte retrieval or even after embryo transfer has been performedb||General|
|3||I want my doctor to practice empathy, instead of only working on the technical or financial part||Staff attitudes|
|4||I want the hospital to have separate waiting rooms for pregnant women and couples being treated for infertility||Organization|
|5||I want to be able to arrange appointments during the daytime as well as in the evenings||Organization|
|General practice care|
|1||I want my gynecologist and GPc to have good communication, so my GP knows what is going on with us||Referral|
|2||I want my GP to make a referral immediately after we have been trying to conceive for a year||Referral|
|3||I want to have my first medical consultation with my gynecologist within 1 month after referral.||Organization|
|4||I want my GP to be informed of possible causes of infertility, in both women and men||General|
|5||I want my GP to pay attention to nonmedical issues, such as stress, anxiety, relational problems, and sexuality||Communication|
|1||I want also to be able to receive treatments on weekends (Saturdays and Sundays)||Organization|
|2||I want all members of the fertility treatment team to apply one policy regarding my infertility treatment||Organization|
|3||I want my gynecologist to inform me of all possible fertility treatment options, even if these are outside the hospital||Information provision|
|4||I want my gynecologist to inform me about the different phases of treatment and their expected time span||Information provision|
|5||I want assisted hatching to be possible or available in the Netherlands||Therapy|
|1||I want my urologist and gynecologist to have good communication||Organization|
|2||I want to be informed of the investigations that are to be performed by the urologist||Examination|
|3||I want to have a permanent urologist who is specialized in infertility||Organization|
|4||I want to have a consultation with a urologist within 1 month after referral||Organization|
|5||I want my urologist to involve my partner in the conversation||Communication|
|1||I want to be informed as soon as possible when our embryos do not divide correctly||Organization|
|2||I want Dutch laboratories to share protocols and learn from each other’s experiences||Organization|
|3||I want to be informed of the causes of nonviability of our frozen embryos, if appropriate||Organization|
a Subsections were derived from the website’s structure and defined by the user.
b Recommendation was excluded, since it fell out of the scope of the clinical practice guideline.
c General practitioner.
|Gender, n (%)|
|Age (years), mean (SD)||35 (5.24)|
|Type of infertility, n (%)|
|Duration of infertility (months), median (range)||36 (0–71)|
|Current phase in fertility care, n (%)|
|No pregnancy after fertility treatment||8 (18%)|
|Pregnant achieved by fertility treatment||4 (9%)|
|Level of education, n (%)|
|Membership in Freya, n (%)||24 (53%)|
|Factor influencing adoption of the wiki|| Proportion disagreeing |
with the factora
|Ease of use of the website|
|Findability of the website||37||82%|
|Accessibility of the website||35||78%|
|Clarity of log-in location on the website||27||60%|
|Clarity on the goal of the website||28||62%|
|Clarity on instructions for using the website||24||53%|
|Efficiency of the website (ie, speed at which the website enabled users to accurately and successfully add and modify recommendations)||24||53%|
|Content of the website|
|Comprehensiveness of the clarifying text on the website||30||66%|
|Satisfaction with the content of the formulated recommendations||25||56%|
|Usefulness of clustering recommendations into sections in searching for existing recommendations||23||51%|
|Similarity between formulated recommendations and participants’ actual opinions on fertility care||23||51%|
|Value of using the wiki|
|Suitability of the wiki for obtaining recommendations for clinical practice guideline development||32||71%|
|Ease of using the wiki||24||53%|
|Accessibility of the wiki||27||60%|
a Number (%) of participants who rated the positively formulated statements on the evaluation factors as disagree (scores 4 or 5).
|Aspect of improvement|| Respondents |
| Sample quotes (translated from Dutch) from |
in-depth interviews (I)
and online questionnaires (Q)
|Usability of the website|
|Findability of the website||10||22%|| Q:Hard to find|
Q: I think it is awkward that the website is only findable through the Freya website
I: I wouldn’t know how to find the website, unless through the Freya website
|Accessibility of the website||2||4%||I: I was unable to find the log-in location or request a new password|
|Content of the website|
|Comprehensiveness of clarifying text||1||2%||Q: Unclear|
|Clearness of description of the goal of the wiki||4||8%|| I: The description is a bit unclear; therefore, I previously thought to check it more precisely, but I still haven’t done this|
I: I had not concluded that the recommendations were directly integrated in a professional guideline
|Clearness of instructions for use||1||2%|
|Satisfaction with formulated recommendations||8||16%||I:...but there are recommendations I am not satisfied with, I would suggest that participants can prioritize recommendations that they are satisfied with in an earlier stage, then you only have to list the most important recommendations in one screen|
|Similarity between actual preferences and recommendations||4||8%|| Q: I would like to see why a specific recommendation was formulated, separately from the recommendation|
I: There are too many recommendations on the website, but there are recommendations I am not satisfied with. I would suggest that participants can prioritize recommendations that they are satisfied with
|Clarity of the structure in which recommendations are placed on the website||30||66%|| I: Structure is good but the provided sections are incomplete, for example the care provided by a psychologist or other forms of mental counseling. Psychosocial concerns are always underestimated in fertility care|
Q: The used structure is good, but for searching an existing recommendation it would be valuable to add a search function to the website
|Relationship between length and number of recommendations and their presentation on one screen||32||71%|| Q: There are too many recommendations on the website|
I: Recommendations are too long, sometimes it’s more like a story, which is very interesting, but I wonder if the doctors are taking this as serious input to a guideline
Q: The prioritization is hard due to the large number of recommendations
|Education provision on the website||19||42%|| Q: It might be valuable if the website provides usable links to high-quality websites|
Q: Information on treatment options might enrich the website
Q: I would like to find information on causal factors of infertility
Q: Practical information about compensations for treatment per insurance company, regional psychological services, plural miscarriages, infertility, and referral
|Characteristics of the wiki|
|Usability of wiki methodology||6||13%|| Q: The website is not user friendly...the number of visible recommendations makes it unclear|
Q: Recommendations given contain too many words
I: I really don’t have a clue about what constitutes a high-quality recommendation
I: It would be valuable to apply an automatic program, through which patients are able to formulate recommendations
|Accessibility of wiki methodology|
|Efficiency of wiki methodology||5||11%|| Q: Prioritizing is hard and not efficient in this stage, the list of recommendations is too long|
I: The efficiency might be improved if you ask patients immediately after formulating a recommendation to prioritize the most important recommendations
|Layout of the website|
|Impression of the layout||33||73%|| Q: Nonattractive/not a modern/not a fashionable website|
Q: The layout is not from today
Q: Looks unprofessional
|Communication with wiki users|
|Marketing||6||13%||Q: This good initiative requires a better marketing approach to reach more participants|
|Community feeling of the wiki||3||6%|| I: More communication on related activities and results will increase the number of patients that will come back|
Q: Effect of the recommendations on the guideline is unclear
a 45 participants completed the online evaluation questionnaire, of whom 3 participated in the in-depth interviews.
In this study, we showed that the wiki is a feasible tool to ensure active patient participation in the development of a Dutch multidisciplinary CPG on infertility. The high numbers of page views (36,473), unique visitors (298), and recommendations formulated (289) implies patients’ willingness and ability to contribute to CPGs through a wiki-based method. We also showed that such a wiki is a useful information source for patients.
Second, we gained a final set of 21 selected recommendations, which were assessed as being eligible to be integrated directly and transparently into the CPG. Third, patients had positive views on the experienced privacy, ease of navigation, divisional structure of the wiki, and its potential befits. A total of 98% of the patients would recommend the website and 84% would participate again in a similar project. This study also provided some important suggestions to improve this participatory tool for patients in the development of CPGs, concerning ease of use, content and layout of the website, and characteristics of the wiki tool.
Comparison With Existing Techniques
Several studies on specialized medical wikis (eg, wikis that fall outside the scope of a general encyclopedia) have been published, but most particularly focus on education of medical students  or collaboration between health care professionals [ , ], rather than on patients, and did not include a process evaluation. Only Gupta and colleagues [ ] and Archambault [ ] involved a group of preselected patients as well as professionals in the development of an asthma action plan through a wiki. However, results are premature, since this study was conducted over a very short time period (weeks), and a wiki needs more time to build content (approximately 7-8 months) [ ]. Furthermore, Gupta and colleagues’ and Archambault’s evaluation of the wiki tool was not focused on patients’ experiences and was less extensive than our multifaceted approach to gaining insight into patients’ barriers to adoption of our wiki. In this study, we involved a large number of patients (298), which cannot be realized using traditional methods, such as focus groups, in which participation is generally restricted to a maximum of 8 participants [ ]. We even assessed the final selection of top recommendations for their eligibility for direct integration into the CPG. Thus, the patients’ contribution to the CPG was clearly illustrated by integrating their recommendations in their entirety, indicated by the new P level (Patients). We also addressed other practical limitations of the methods used to enhance patient participation in CPG development, such as organizational (eg, recruitment of participants), financial (eg, travel costs), and sociopolitical (eg, professional resistance to including patients in CPG development group) constraints.
Professionals and patients’ representatives could also use the wiki and had the opportunity to informed themselves about patients’ views and to bring up content for discussion in the CPG development group. According to the results of the evaluation questionnaire and the interviews, this content was also helpful to patients as an information source, which may also explain the relatively large number of page views. Although providing information was not the initial goal of this wiki, its relevance is in agreement with published literature on conventional wikis  and with European patients’ perception of the importance of the Internet as a source of information [ ]. Hence, this unintended consequence concurrently yields challenges for improvement and might be aided by providing clearer instructions for use and description of the goal of the wiki, but also addresses important implications for future studies in this field. Next to the informational value of formulated recommendations for high-quality care, attention should be paid to useful links to relevant websites that may potentially attract more patients to the wiki website and increase the chances for adoption of an improved version of the wiki.
Although drawbacks to active patient participation methods were reduced, this study drew attention to some other potential implications derived from patients’ suggestions that might improve the use of a future medical dedicated wiki exclusively for actively involving patients in CPG development. First, structuring recommendations and limiting the number and length of recommendations to presentation on one screen may improve usability . Second, using a fixed format in the formulation of recommendations, based on relevant items of the GLIA instrument, may not only improve usability and accessibility of the wiki [ , ], but may also improve the efficiency of the wiki and the usefulness of recommendations in being integrated directly into the CPG. Introducing a motivation page might give patients the opportunity to add a personal touch to the recommendation. Third, a prioritization system, continuously refining the similarity between patients’ perspectives and the top5 recommendations (eg, by rating recommendations after every contribution), could improve the tools’ efficiency by avoiding separate prioritization of recommendations and could improve patients’ satisfaction with the highest-rated recommendations. This modality would also allow more flexible use by CPG developers at the time of their choosing. In addition to the suggested modalities, some known refinements in overall usability (eg, findability, prominent log-in location), content (comprehensiveness of text), and layout of the website might improve use of the wiki and would be reduced by repeated cycles of design, evaluation, and redesign [ , ]. Furthermore, a user-centered design, in which patients codevelop such new modalities, may improve future implementability and provide chances for local adaptation of a redesigned wiki website [ , ]. Both the feasibility of a wiki as a participatory tool for patients in the development of CPGs and the recommendations for future wiki-based initiatives illustrate the value of eHealth. With this in mind, numerous participatory applications based on wikis are conceivable and may be valuable in various fields of research. In the field of guideline development, guideline-derived initiatives actively involving patients in the development of patient information leaflets or treatment action plans, in addition to fully online-based CPGs, may also benefit from our results. Finally, our results add to the knowledge base about wikis in health care [ ].
This wiki has been tested in the field of infertility care, representing a relatively young target group . More than 98% of this group use the Internet [ ]. This participant characteristic is associated with more frequent health-related Internet use [ - ]. Therefore, the participants in our study were an ideal subgroup for testing and evaluating a wiki-based method, which argues against the generalizability of our findings to other patient groups. Nevertheless, health-related Internet use in Europe is increasing over time [ ]. Hence, it seems to be a question of time until older people or their caregivers, or both, will be using such tools [ ]. Furthermore, this feasibility study provided an important exploratory evaluation component, which resulted in valuable information for future studies in this field but also had certain limitations. First, based on the results of a recent systematic review from Gagnon and colleagues [ ], we acknowledge that the items used in our evaluation questionnaire might be incomplete. However, the results of our study add to those from the limited number of previously published studies on patients’ facilitators of and barriers to adoption of eHealth applications [ , ]. Second, the heuristics used were not based on a validated questionnaire and were too limited for drawing conclusions on the usability that patients perceived. Therefore, a next step in future development of a wiki-based participatory tool for patients in CPG development should be to include a broader evaluation of the potentially influencing factors on adoption, including more organizational factors and a heuristic evaluation.
Third, the participation rate in the evaluation of the wiki might have subjected our study to a participation bias of potentially the most motivated wiki users. However, this is a known limitation in the active use of wikis in general: the most motivated users provide most of the content . Finally, this feasibility study did not assess the representativeness of either the participants or the final set of recommendations in the wiki.
The wiki is a promising and feasible tool to actively involve patients in CPG development. To improve the tool’s ease of use and practical aspects to enhance direct integration of recommendations into the CPG, a more specialized and refined wiki should be developed. This should include new modalities, such as automatically shortening the number and length of recommendations, using a fixed format for formulation of recommendations, using a continuous prioritization system for selection of the most important recommendations, and including a separate motivation page. Furthermore, in the development, attention should be paid to the informational character of such a wiki. To improve future implementability, a modified tool should preferably be codeveloped and evaluated by patients in a user-centered design study. Furthermore, representativeness of patients and recommendations should be integrated into this next phase.
The authors thank all participants of FreyaWIKI.
Funding: This project was funded by the Netherlands Organization for Health Research and Development (ZonMW), grant no. 150020015, The Hague. This open access publication was funded by the Dutch Organization for Scientific Research (NWO), grant no. 036.001.927. Both funders did not have any role in the design and conduct of the study, the collection, management, analysis, and interpretation of data, or in the preparation, review, or approval of the manuscript.
All researchers are independent from the funders.
Previous presentation/report: The content of this paper has not been copyrighted, published, or presented previously and is not under consideration for publication elsewhere.
Ethics approval: In this study, ethics approval was not required. However, all participants gave informed consent before taking part. All observational data and transcripts of interviews were anonymized.
EB was guarantor, designed the study, modified the recommendations, conducted and (qualitatively) analyzed the online evaluation questionnaire and in-depth interviews, interpreted the analysis of the use of the wiki and its evaluation, and drafted the paper. WN designed the study, checked the analysis, interpreted the analysis, and drafted the paper. JK designed the study, modified the recommendations, interpreted the analysis, and drafted the paper. JB designed the study, interpreted the analysis, and drafted the paper. RH designed the study, interpreted the analysis, and drafted the paper. JAMK led the research team, designed the study, was principal investigator, interpreted the analysis, and drafted the paper. All authors reviewed consecutive drafts of the paper and gave their final approval of the version to be published.
Conflicts of Interest
Multimedia Appendix 1
Constructs of the online evaluation questionnaire.PDF File (Adobe PDF File), 50KB
Multimedia Appendix 2
Screenshot of recommendations on FreyaWIKI.PDF File (Adobe PDF File), 221KB
Multimedia Appendix 3
Screenshot of the FreyaWIKI homepage.PDF File (Adobe PDF File), 233KB
- Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst Rev 2006(3):CD004563. [CrossRef] [Medline]
- Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. In: Graham RL, Mancher M, Wolman DM, Greenfield S, Steinberg E, editors. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011.
- van Wersch A, Eccles M. Involvement of consumers in the development of evidence based clinical guidelines: practical experiences from the North of England evidence based guideline development programme. Qual Health Care 2001 Mar;10(1):10-16 [FREE Full text] [Medline]
- Crawford MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, et al. Systematic review of involving patients in the planning and development of health care. BMJ 2002 Nov 30;325(7375):1263. [Medline]
- Burgers JS, Fervers B, Haugh M, Brouwers M, Browman G, Philip T, et al. International assessment of the quality of clinical practice guidelines in oncology using the Appraisal of Guidelines and Research and Evaluation Instrument. J Clin Oncol 2004 May 15;22(10):2000-2007. [CrossRef] [Medline]
- Bastian H. Raising the standard: practice guidelines and consumer participation. Int J Qual Health Care 1996 Oct;8(5):485-490 [FREE Full text] [Medline]
- Gagliardi AR, Brouwers MC, Palda VA, Lemieux-Charles L, Grimshaw JM. How can we improve guideline use? A conceptual framework of implementability. Implement Sci 2011;6:26 [FREE Full text] [CrossRef] [Medline]
- AGREE Enterprise. Appraisal of Guidelines for Research and Evaluation (AGREE Instrument) URL: http://www.agreetrust.org/mod_sys/errors/error.html?aspxerrorpath=/index.aspx [accessed 2012-02-16] [WebCite Cache]
- Schünemann HJ, Fretheim A, Oxman AD. Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement. Health Res Policy Syst 2006;4:22 [FREE Full text] [CrossRef] [Medline]
- Rogers WA. Are guidelines ethical? Some considerations for general practice. Br J Gen Pract 2002 Aug;52(481):663-668. [Medline]
- National Institute for Health and Clinical Excellence. 2009 Apr 03. The Guidelines Manual 2009 URL: http://www.nice.org.uk/aboutnice/howwework/developingniceclinicalguidelines/clinicalguidelinedevelopmentmethods/GuidelinesManual2009.jsp [accessed 2012-10-01] [WebCite Cache]
- Healthcare Improvement Scotland. Scottish Intercollegiate Guidelines Network (SIGN). 2011 Nov. SIGN 50: A Guideline Developer's Handbook, Revised Edition URL: http://www.sign.ac.uk/pdf/sign50.pdf [accessed 2012-02-16] [WebCite Cache]
- Díaz Del Campo P, Gracia J, Blasco JA, Andradas E. A strategy for patient involvement in clinical practice guidelines: methodological approaches. BMJ Qual Saf 2011 Sep;20(9):779-784. [CrossRef] [Medline]
- Légaré F, Boivin A, van der Weijden T, Pakenham C, Burgers J, Légaré J, et al. Patient and public involvement in clinical practice guidelines: a knowledge synthesis of existing programs. Med Decis Making 2011;31(6):E45-E74. [CrossRef] [Medline]
- Boivin A, Currie K, Fervers B, Gracia J, James M, Marshall C, G-I-N PUBLIC. Patient and public involvement in clinical guidelines: international experiences and future perspectives. Qual Saf Health Care 2010 Oct;19(5):e22. [CrossRef] [Medline]
- Krahn M, Naglie G. The next step in guideline development: incorporating patient preferences. JAMA 2008 Jul 23;300(4):436-438. [CrossRef] [Medline]
- Cross M. How the internet is changing health care. BMJ 2008;337:a883. [Medline]
- Giustini D. How Web 2.0 is changing medicine. BMJ 2006 Dec 23;333(7582):1283-1284. [CrossRef] [Medline]
- Heilman JM, Kemmann E, Bonert M, Chatterjee A, Ragar B, Beards GM, et al. Wikipedia: a key tool for global public health promotion. J Med Internet Res 2011;13(1):e14 [FREE Full text] [CrossRef] [Medline]
- Trevena L. WikiProject medicine. BMJ 2011;342:d3387. [Medline]
- Haagen EC, Tuil W, Hendriks J, de Bruijn RP, Braat DD, Kremer JA. Current Internet use and preferences of IVF and ICSI patients. Hum Reprod 2003 Oct;18(10):2073-2078 [FREE Full text] [Medline]
- Weissman A, Gotlieb L, Ward S, Greenblatt E, Casper RF. Use of the internet by infertile couples. Fertil Steril 2000 Jun;73(6):1179-1182. [Medline]
- Wischmann T. Implications of psychosocial support in infertility--a critical appraisal. J Psychosom Obstet Gynaecol 2008 Jun;29(2):83-90. [CrossRef] [Medline]
- Gnoth C, Godehardt E, Frank-Herrmann P, Friol K, Tigges J, Freundl G. Definition and prevalence of subfertility and infertility. Hum Reprod 2005 May;20(5):1144-1147 [FREE Full text] [CrossRef] [Medline]
- Taylor A. ABC of subfertility: extent of the problem. BMJ 2003 Aug 23;327(7412):434-436. [CrossRef] [Medline]
- Nachtigall RD. International disparities in access to infertility services. Fertil Steril 2006 Apr;85(4):871-875. [CrossRef] [Medline]
- UMC St Radboud. 2008. FreyaWIKI URL: https://www.freyawiki.nl/ [accessed 2012-02-16] [WebCite Cache]
- Zegers-Hochschild F, Adamson GD, de Mouzon J, Ishihara O, Mansour R, Nygren K, International Committee for Monitoring Assisted Reproductive Technology, World Health Organization. International Committee for Monitoring Assisted Reproductive Technology (ICMART) and the World Health Organization (WHO) revised glossary of ART terminology, 2009. Fertil Steril 2009 Nov;92(5):1520-1524. [CrossRef] [Medline]
- Dancet EA, Nelen WL, Sermeus W, De Leeuw L, Kremer JA, D'Hooghe TM. The patients' perspective on fertility care: a systematic review. Hum Reprod Update 2010;16(5):467-487 [FREE Full text] [CrossRef] [Medline]
- Shiffman RN, Dixon J, Brandt C, Essaihi A, Hsiao A, Michel G, et al. The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak 2005;5:23 [FREE Full text] [CrossRef] [Medline]
- Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol 2010;10:67 [FREE Full text] [CrossRef] [Medline]
- Yusof MM, Kuljis J, Papazafeiropoulou A, Stergioulas LK. An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform 2008 Jun;77(6):386-398. [CrossRef] [Medline]
- Francis JJ, Johnston M, Robertson C, Glidewell L, Entwistle V, Eccles MP, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health 2010 Dec;25(10):1229-1245. [CrossRef] [Medline]
- Silverman D. Doing Qualitative Research. 3rd edition. London: Sage Publications Ltd; 2009.
- Phillips B, Ball C, Sackett D, Badenoch D, Straus S, Haynes B, et al. Oxford Centre for Evidence Based Medicine. 2009 Mar. Levels of Evidence URL: http://www.cebm.net/index.aspx?o=4590 [accessed 2012-06-21] [WebCite Cache]
- Jalali A, Mioduszewski M, Gauthier M, Varpio L. Wiki use and challenges in undergraduate medical education. Med Educ 2009 Nov;43(11):1117. [CrossRef] [Medline]
- Naik AD, Singh H. Electronic health records to coordinate decision making for complex patients: what can we learn from wiki? Med Decis Making 2010;30(6):722-731. [CrossRef] [Medline]
- Archambault PM, Légaré F, Lavoie A, Gagnon MP, Lapointe J, St-Jacques S, et al. Healthcare professionals' intentions to use wiki-based reminders to promote best practices in trauma care: a survey protocol. Implement Sci 2010;5:45 [FREE Full text] [CrossRef] [Medline]
- Gupta S, Wan FT, Newton D, Bhattacharyya OK, Chignell MH, Straus SE. WikiBuild: a new online collaboration process for multistakeholder tool development and consensus building. J Med Internet Res 2011;13(4):e108 [FREE Full text] [CrossRef] [Medline]
- Archambault PM. WikiBuild: a new application to support patient and health care professional involvement in the development of patient support tools. J Med Internet Res 2011;13(4):e114 [FREE Full text] [CrossRef] [Medline]
- Hansen DL. Knowledge Sharing, Maintenance, and Use in Online Support Communities [doctoral dissertation]. Ann Arbor, MI: University of Michigan; 2007. URL: http://deepblue.lib.umich.edu/bitstream/2027.42/57608/2/shakmatt_1.pdf [accessed 2012-06-21] [WebCite Cache]
- Pope C, van Royen P, Baker R. Qualitative methods in research on healthcare quality. Qual Saf Health Care 2002 Jun;11(2):148-152 [FREE Full text] [Medline]
- Kummervold PE, Chronaki CE, Lausen B, Prokosch HU, Rasmussen J, Santana S, et al. eHealth trends in Europe 2005-2007: a population-based survey. J Med Internet Res 2008;10(4):e42 [FREE Full text] [CrossRef] [Medline]
- Spyridakis JH. Guidelines for authoring comprehensible Web pages and evaluating their success. Tech Commun 2000 Aug 01;47(3):359-382 [FREE Full text]
- Gustafson DH, Wyatt JC. Evaluation of ehealth systems and services. BMJ 2004 May 15;328(7449):1150. [CrossRef] [Medline]
- Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Comput Biol Med 2002 May;32(3):141-149. [Medline]
- Taylor HA, Sullivan D, Mullen C, Johnson CM. Implementation of a user-centered framework in the development of a web-based health information database and call center. J Biomed Inform 2011 Oct;44(5):897-908. [CrossRef] [Medline]
- Roehrer E, Cummings E, Ellis L, Turner P. The role of user-centred design within online community development. Stud Health Technol Inform 2011;164:256-260. [Medline]
- Archambault PM, van de Belt TH, Grajales FJ, Eysenbach G, Aubin K, Gold I, et al. Wikis and collaborative writing applications in health care: a scoping review protocol. JMIR Res Protoc 2012;1(1):e1. [CrossRef]
- Brandes M, Hamilton CJ, Bergevoet KA, de Bruin JP, Nelen WL, Kremer JA. Origin of multiple pregnancies in a subfertile population. Acta Obstet Gynecol Scand 2010 Sep;89(9):1149-1154. [CrossRef] [Medline]
- Atkinson NL, Saperstein SL, Pleis J. Using the internet for health-related activities: findings from a national probability sample. J Med Internet Res 2009;11(1):e4 [FREE Full text] [CrossRef] [Medline]
- Baker L, Wagner TH, Singer S, Bundorf MK. Use of the Internet and e-mail for health care information: results from a national survey. JAMA 2003 May 14;289(18):2400-2406. [CrossRef] [Medline]
- Cotten SR, Gupta SS. Characteristics of online and offline health information seekers and factors that discriminate between them. Soc Sci Med 2004 Nov;59(9):1795-1806. [CrossRef] [Medline]
- Andreassen HK, Bujnowska-Fedak MM, Chronaki CE, Dumitru RC, Pudule I, Santana S, et al. European citizens' use of E-health services: a study of seven countries. BMC Public Health 2007;7:53 [FREE Full text] [CrossRef] [Medline]
- Karunanithi M. Monitoring technology for the elderly patient. Expert Rev Med Devices 2007 Mar;4(2):267-277. [CrossRef] [Medline]
- Gagnon MP, Desmartis M, Labrecque M, Car J, Pagliari C, Pluye P, et al. Systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals. J Med Syst 2012 Feb;36(1):241-277. [CrossRef] [Medline]
- Chhanabhai P, Holt A. Consumers are ready to accept the transition to online and electronic records if they can be assured of the security measures. MedGenMed 2007;9(1):8 [FREE Full text] [Medline]
- Gagnon MP, Shaw N, Sicotte C, Mathieu L, Leduc Y, Duplantie J, et al. Users' perspectives of barriers and facilitators to implementing EHR in Canada: a study protocol. Implement Sci 2009;4:20 [FREE Full text] [CrossRef] [Medline]
|AGREE: Appraisal of Guidelines Research and Evaluation|
|CPG: clinical practice guideline|
|GLIA: Guideline Implementability Appraisal|
Edited by G Eysenbach; submitted 16.02.12; peer-reviewed by N Nijland, P Archambault; comments to author 14.04.12; revised version received 22.06.12; accepted 26.07.12; published 26.10.12
©Elvira ME den Breejen, Willianne LDM Nelen, Jose M.L. Knijnenburg, Jako S Burgers, Rosella PMG Hermens, Jan AM Kremer. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 26.10.2012.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.