Development and Evaluation of a New Serious Game for Continuing Medical Education of General Practitioners (Hygie): Double-Blinded Randomized Controlled Trial

Background Continuing medical education is important but time-consuming for general practitioners (GPs). Current learning approaches are limited and lack the ability to engage some practitioners. Serious games are new learning approaches that use video games as engaging teaching material. They have significant advantages in terms of efficiency and dissemination. Objective The aim of this study was to create a serious game and to evaluate it in terms of effectiveness and satisfaction, comparing it with a traditional method of continuing education—article reading. Methods We produced a prototype video game called Hygie on the 5 most common reasons of consultation in general practice using 9 articles from independent evidence-based medicine journals (reviews from Prescrire and Minerva). We created 51 clinical cases. We then conducted a double-blinded randomized trial comparing the learning provided by a week of access to the game versus source articles. Participants were GPs involved as resident supervisors in 14 French university departments of family practice, recruited by email. Primary outcomes were (1) mean final knowledge score completed 3 to 5 weeks after the end of the intervention and (2) mean difference between knowledge pretest (before intervention) and posttest (3 to 5 weeks after intervention) scores, both scaled on 10 points. Secondary outcomes were transfer of knowledge learned to practice, satisfaction, and time spent playing. Results A total of 269 GPs agreed to participate in the study. Characteristics of participants were similar between learning groups. There was no difference between groups on the mean score of the final knowledge test, with scores of 4.9 (95% CI 4.6-5.2) in the Hygie group and 4.6 (95% CI 4.2-4.9) in the reading group (P=.21). There was a mean difference score between knowledge pre- and posttests, with significantly superior performance for Hygie (mean gain of 1.6 in the Hygie group and 0.9 in the reading group; P=.02), demonstrating a more efficient and persistent learning with Hygie. The rate of participants that reported to have used the knowledge they learned through the teaching material was significantly superior in the Hygie group: 77% (47/61) in the Hygie group and 53% (25/47) in the reading group; odds ratio 2.9, 95% CI 1.2-7.4. Moreover, 87% of the opinions were favorable, indicating that Hygie is of interest for updating medical knowledge. Qualitative data showed that learners enjoyed Hygie especially for its playful, interactive, and stimulating aspects. Conclusions We conclude that Hygie can diversify the offering for continuing education for GPs in an effective, pleasant, and evidence-based way. Trial Registration ClinicalTrials.gov NCT03486275; https://clinicaltrials.gov/ct2/show/NCT03486275


1a-i) Identify the mode of delivery in the title
Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internetbased" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if offline products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms.
Does your paper address subitem 1a-i? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "serious game" subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential

1a-ii) Non-web-based components or important co-interventions in title
Mention non-web-based components or important co-interventions in title, if any (e.g., "with telephone support").
Does your paper address subitem 1a-ii?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : no co-intervention 1a-iii) Primary condition or target group in the title Mention primary condition or target group in the title, if any (e.g., "for children with Type I Diabetes") Example: A Web-based and Mobile Intervention with Telephone Support for Children with Type I Diabetes: Randomized Controlled Trial Does your paper address subitem 1a-iii? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "general practitioners" 1b) ABSTRACT: Structured summary of trial design, methods, results, and conclusions NPT extension: Description of experimental treatment, comparator, care providers, centers, and blinding status.
subitem not at all important 1 2 3 4 5 essential 1b-i) Key features/functionalities/components of the intervention and comparator in the METHODS section of the ABSTRACT Mention key features/functionalities/components of the intervention and comparator in the abstract. If possible, also mention theories and principles used for designing the site. Keep in mind the needs of systematic reviewers and indexers by including important synonyms. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) Does your paper address subitem 1b-i? * Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "We produced a prototype video game called Hygie on the 5 most common reasons of consultation in general practice using 9 articles from independent evidencebased medicine journals (reviews from Prescrire and Minerva). We created 51 clinical cases." subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential 1b-ii) Level of human involvement in the METHODS section of the ABSTRACT Does your paper address subitem 1b-ii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Participants were GPs involved as resident supervisors in 14 French university departments of family practice" 1b-iii) Open vs. closed, web-based (self-assessment) vs. face-to-face assessments in the METHODS section of the ABSTRACT Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic or a closed online user group (closed usergroup trial), and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment). Clearly say if outcomes were self-assessed through questionnaires (as common in web-based trials Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "recruited by email" 1b-iv) RESULTS section in abstract must contain use data Report number of participants enrolled/assessed in each group, the use/uptake of the intervention (e.g., attrition/adherence metrics, use over time, number of logins etc.), in addition to primary/secondary outcomes. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) Does your paper address subitem 1b-iv?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "New teaching materials called "serious games" are efficient (6,7) and easily disseminated methods for education (8). Indeed, they offer the possibility of combining learning activities such as testing (9), feedback (10), spaced repetition (11), problem-based learning (12,13) with a positive experience. Learning challenges can be provided by these games (14,15) in a risk-free environment (16). Therefore, serious games give active participation and autonomy to the learner, both of which are crucial qualities in adult education (17).
Few serious games have been developed with the goal of facilitating continuous medical education for health professionals (18) and general (19,20). To our knowledge, no existing game covers topics related to family medicine." Does your paper address CONSORT subitem 2b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "The aim of this work was to develop a prototype of a new serious game called Hygie for continuing medical education for the general practitioner and to assess its efficiency and user acceptance as compared with a traditional activity (article reading) in a randomized trial." 3a) Description of trial design (such as parallel, factorial) including allocation ratio 3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 3a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "They were randomized using the rand function of PHP language (allocation ratio 1:1), to either the intervention group (access online to Hygie for one week) or to the control group (access online to the 9 articles)" Does your paper address CONSORT subitem 3b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Not applicable : no change

3b-i) Bug fixes, Downtimes, Content Changes
Bug fixes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug fixes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have influenced study design such as staff changes, system failures/downtimes, etc. [2].

4a) Eligibility criteria for participants
subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 3b-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : bugs were fixed using a pilot study Does your paper address CONSORT subitem 4a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "We asked all 35 French University departments of general practice to contact GPs involved as resident supervisors by email"

4a-i) Computer / Internet literacy
Computer / Internet literacy is often an implicit "de facto" eligibility criterion -this should be explicitly clarified.
Does your paper address subitem 4a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not relevant : recruitment by email subitem not at all important 1 2 3 4 5 essential 4a-ii) Open vs. closed, web-based vs. face-to-face assessments: Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic, and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant.
In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email confirmation, phone calls) were used to detect/prevent these.

Does your paper address subitem 4a-ii? *
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "We asked all 35 French University departments of general practice to contact GPs involved as resident supervisors by e-mail to participate in a real-life experience learning where they would have access to a e-learning support for seven days, without mentioning the nature of the educational support.
[...] After agreeing to participate, GPs accessed the study website where they completed [...] Participants did not know if they were assigned to the intervention or control group and did not know which intervention was performed in the other group. " subitem not at all important 1 2 3 4 5 essential 4b) Settings and locations where the data were collected

4a-iii) Information giving during recruitment
Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results.
Does your paper address subitem 4a-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Information was delivered to participants about the purpose, the duration, the time to devote to the study, and anonymization of results." Does your paper address CONSORT subitem 4b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Data were collected on a Structured Query Language database." subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential 4b-i) Report if outcomes were (self-)assessed through online questionnaires Clearly report if outcomes were (self-)assessed through online questionnaires (as common in web-based trials) or otherwise.
Does your paper address subitem 4b-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "After 3 weeks without access to the teaching materials, participants received a final, 20-item knowledge questionnaire" 4b-ii) Report how institutional affiliations are displayed Report how institutional affiliations are displayed to potential participants [on ehealth media], as affiliations with prestigious hospitals or universities may affect volunteer rates, use, and reactions with regards to an intervention.(Not a required item -describe only if this may bias results) Does your paper address subitem 4b-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Institutional affiliations of the investigators were indicated at the end of the email." 5) The interventions for each group with sufficient details to allow replication, including how and when they were actually administered subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential 5-i) Mention names, credential, affiliations of the developers, sponsors, and owners Mention names, credential, affiliations of the developers, sponsors, and owners [6] (if authors/evaluators are owners or developer of the software, this needs to be declared in a "Conflict of interest" section or mentioned elsewhere in the manuscript).

Does your paper address subitem 5-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Louis-Baptiste Jaunay is the owner of hygie-jeu.fr website. No other COI declared."

5-ii) Describe the history/development process
Describe the history/development process of the application and previous formative evaluations (e.g., focus groups, usability testing), as these will have an impact on adoption/use rates and help with interpreting results.

Does your paper address subitem 5-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Design and Development of Hygie We produced a prototype video game called Hygie in which the player is a GP in the process of treating several patients. We defined topics for this prototype based on the 5 most frequent reasons for consultation in France [21]: hypertension, health check and prevention, dyslipidemia, acute fever, and rhinopharyngitis. For these 5 topics, we reviewed 9 articles in 2 French evidence-based journals: 6 from Prescrire and 3 from Minerva [22][23][24][25][26][27][28][29][30]. We selected these 2 journals because they provide robust evidence-based recommendations and are strictly independent from industrial and institutional influences. From these 9 articles, we created 51 short clinical cases, each having 1 question that could be answered either by multiple choice or free text. The game was coded using HTML 5, Cascading Style Sheets 3, JavaScript (ECMAScript 2015), and Hypertext Preprocessor (PHP) 7. Graphics were created using Adobe Illustrator and Adobe Photoshop (Figure 1). Learning methods incorporated into the game included statement of educational objectives, immersion in a general medical consultation setting, problem-based learning with active restitution of knowledge, spaced recall, stimulation of intrinsic motivation by earning points, and having goals and levels with a "final boss" for each level. Humoristic elements such as puns in patients' names were included to maximize engagement. A preliminary test phase was conducted with 11 GPs and 9 residents in general practice. The preliminary test allowed us to detect and solve bugs, clarify questions, and sort questions into 5 levels of difficulty." subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential

5-iii) Revisions and updating
Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was "frozen" during the trial. Describe dynamic components such as news feeds or changing content which may have an impact on the replicability of the intervention (for unexpected events see item 3b).

Does your paper address subitem 5-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "A preliminary test phase was conducted with 11 GPs and 9 residents in general practice. The preliminary test allowed us to detect and solve bugs, clarify questions, and sort questions into 5 levels of difficulty."

5-iv) Quality assurance methods
Provide information on quality assurance methods to ensure accuracy and quality of information provided [1], if applicable.
subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 5-iv?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "For these 5 topics, we reviewed 9 articles in 2 French evidence-based journals: 6 from Prescrire and 3 from Minerva [22][23][24][25][26][27][28][29][30]. We selected these 2 journals because they provide robust evidence-based recommendations and are strictly independent from industrial and institutional influences." 5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing flowcharts of the algorithms used Ensure replicability by publishing the source code, and/or providing screenshots/screencapture video, and/or providing flowcharts of the algorithms used. Replicability (i.e., other researchers should in principle be able to replicate the study) is a hallmark of scientific reporting.
Does your paper address subitem 5-v?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

5-vi) Digital preservation
Digital preservation: Provide the URL of the application, but as the intervention is likely to change or disappear over the course of the years; also make sure the intervention is archived (Internet Archive, webcitation.org, and/or publishing the source code or screenshots/videos alongside the article). As pages behind login screens cannot be archived, consider creating demo pages which are accessible without login.

Does your paper address subitem 5-vi?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "The prototype of the game is freely accessible on the Web [31]."

5-vii) Access
Access: Describe how participants accessed the application, in what setting/context, if they had to pay (or were paid) or not, whether they had to be a member of specific group. If known, describe how participants obtained "access to the platform and Internet" [1]. To ensure access for editors/reviewers/readers, consider to provide a "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes, see vi). 1 2 3 4 5 essential Does your paper address subitem 5-vii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "After agreeing to participate, GPs accessed the study website where they completed a demographic questionnaire, a knowledge pretest of 5 questions on each of the 5 reasons for consultation. They were randomized using the rand function of PHP language (allocation ratio 1:1) to either the intervention group (Web access to Hygie for 1 week) or to the control group (access online to the 9 articles). Participants had an individual login, allowing them to access only the teaching material assigned to them. They did not know if they were assigned to the intervention or control group and did not know which intervention was performed in the other group." subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 5-viii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Learning methods incorporated into the game included statement of educational objectives, immersion in a general medical consultation setting, problem-based learning with active restitution of knowledge, spaced recall, stimulation of intrinsic motivation by earning points, and having goals and levels with a "final boss" for each level. Humoristic elements such as puns in patients' names were included to maximize engagement."

5-ix) Describe use parameters
Describe use parameters (e.g., intended "doses" and optimal timing for use). Clarify what instructions or recommendations were given to the user, e.g., regarding timing, frequency, heaviness of use, if any, or was the intervention used ad libitum.
Does your paper address subitem 5-ix?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "After 1 week of free access to their respective teaching material (serious game Hygie vs articles), access was terminated." subitem not at all important

5-x) Clarify the level of human involvement
Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21 -generalizability).

Does your paper address subitem 5-x?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : no human involvement

5-xi) Report any prompts/reminders used
Report any prompts/reminders used: Clarify if there were prompts (letters, emails, phone calls, SMS) to use the application, what triggered them, frequency etc. It may be necessary to distinguish between the level of prompts/reminders required for the trial, and the level of prompts/reminders for a routine application outside of a RCT setting (discuss under item 21generalizability).
subitem not at all important 1 2 3 4 5 essential 6a) Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed Does your paper address subitem 5-xi? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Reminders were sent to the 2 groups within 3 and 6 days of access to teaching material"

5-xii) Describe any co-interventions (incl. training/support)
Describe any co-interventions (incl. training/support): Clearly state any interventions that are provided in addition to the targeted eHealth intervention, as ehealth intervention may not be designed as stand-alone intervention. This includes training sessions and support [1]. It may be necessary to distinguish between the level of training required for the trial, and the level of training for a routine application outside of a RCT setting (discuss under item 21generalizability. Does your paper address subitem 5-xii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not appicable: no cointervention Does your paper address CONSORT subitem 6a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Primary outcomes were (1) dynamic and (2) static knowledge assessed by questionnaires: 1. A Dynamic Questionnaire-5 (DQ-5) with 5 items compared individual change of score between pretest (before intervention) and post-test (3-5 weeks after intervention). It was a 5-item questionnaire with each of the 5 questions weighted by a scale ranging from 1 to 3 according to its importance for practice and a global score from 0 to 14. The goal of this questionnaire was to assess progression of each participants. For a simpler interpretation, we scaled the DQ-5 score to be out of 10 rather than 14. 2. A Static Questionnaire-20 (SQ-20) measured mean final score 3 to 5 weeks after intervention. It was a 20-item questionnaire (5 of the dynamic questionnaire plus 15 other questions), with each of the 20 questions weighted by a scale ranging from 1 to 3 according to its importance for practice and a global score from 0 to 58. The goal of this questionnaire was to compare groups, minimizing the potential carryover effects induced by the pretest questionnaire. Like for the DQ-5, for interpretation, we rescaled the SQ-20 to a 0 to 10 scale (rather than 0-58).
Here is an example of a knowledge question that appears in both dynamic and static questionnaires and the scoring method: Question: Which cholesterol-lowering drugs have shown a decrease in mortality and morbidity? Expected answers (free text): pravastatin, simvastatin. Scoring method: It was rated 3 points out of 14: If the 2 right molecules (pravastatin and simvastatin) are mentioned: 3 points If 1 good molecule among pravastatin and simvastatin is mentioned: 1 point In all other cases: 0 points. The 2 knowledge questionnaires and their scale were written from the source articles by 3 experienced physicians who had no information about the game content, with instructions to identify practice-relevant issues in the articles. Participants' questionnaires were scored blindly by a physician not involved in the other stages of the study. Secondary outcomes were (1) the use in medical practice of the knowledge acquired through the teaching material assessed at the time of the final questionnaire (participants answered the question "In the course of your practice, did you use the knowledge you learned through the teaching material?"), (2) time spent playing by participants assigned to Hygie, and (3)

Does your paper address subitem 6a-i?
Copy and paste relevant sections from manuscript text "The 2 knowledge questionnaires and their scale were written from the source articles by 3 experienced physicians who had no information about the game content, with instructions to identify practice-relevant issues in the articles. Participants' questionnaires were scored blindly by a physician not involved in the other stages of the study." 6a-ii) Describe whether and how "use" (including intensity of use/dosage) was defined/measured/monitored Describe whether and how "use" (including intensity of use/dosage) was defined/measured/monitored (logins, logfile analysis, etc.). Use/adoption metrics are important process outcomes that should be reported in any ehealth trial.
subitem not at all important 1 2 3 4 5 essential 6b) Any changes to trial outcomes after the trial commenced, with reasons Does your paper address subitem 6a-ii?
Copy and paste relevant sections from manuscript text "The average total time spent on the Hygie game was measured via server usage data. Average total time spent on the articles was not collected because participants could download the articles and read it offline." 6a-iii) Describe whether, how, and when qualitative feedback from participants was obtained Describe whether, how, and when qualitative feedback from participants was obtained (e.g., through emails, feedback forms, interviews, focus groups).

Does your paper address subitem 6a-iii?
Copy and paste relevant sections from manuscript text "The satisfaction questionnaire, composed of 8 questions and completed at the end of the 1-week learning period, included quantitative and qualitative data about participant satisfaction, time reported as spent on the materials, and additional demographic data (eg, workplace and usual training materials for continuing education). Qualitative data were analyzed by content for themes related to effective learning as well as to illuminate potential strengths and weaknesses of Hygie." 7a) How sample size was determined NPT: When applicable, details of whether and how the clustering by care provides or centers was addressed subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 6b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable: no change 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size Describe whether and how expected attrition was taken into account when calculating the sample size.

Does your paper address subitem 7a-i?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "The number of participants required with 80% power (1−beta) and 5% type I error was estimated before the study. A total of 128 participants were needed to detect a difference of 2 points out of 10 between the groups on the final questionnaire, assuming that the participants in the Hygie group had a final score of 8 out of 10 on average." 7b) When applicable, explanation of any interim analyses and stopping guidelines 8a) Method used to generate the random allocation sequence NPT: When applicable, how care providers were allocated to each trial group 8b) Type of randomisation; details of any restriction (such as blocking and block size) Does your paper address CONSORT subitem 7b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : no interim analyses nor stopping guidelines Does your paper address CONSORT subitem 8a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "They were randomized using the rand function of PHP language (allocation ratio 1:1) to either the intervention group (Web access to Hygie for 1 week) or to the control group (access online to the 9 articles)." 9) Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned 10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions Does your paper address CONSORT subitem 8b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : no restriction Does your paper address CONSORT subitem 9? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : no allocation sequence Does your paper address CONSORT subitem 10? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : no allocation sequence 11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment subitem not at all important 1 2 3 4 5 essential 11a-i) Specify who was blinded, and who wasn't Specify who was blinded, and who wasn't. Usually, in web-based trials it is not possible to blind the participants [1, 3] (this should be clearly acknowledged), but it may be possible to blind outcome assessors, those doing data analysis or those administering co-interventions (if any).

Does your paper address subitem 11a-i? *
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "They did not know if they were assigned to the intervention or control group and did not know which intervention was performed in the other group [...] Participants' questionnaires were scored blindly by a physician not involved in the other stages of the study." subitem not at all important 1 2 3 4 5 essential 11b) If relevant, description of the similarity of interventions (this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention) 11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator" Informed consent procedures (4a-ii) can create biases and certain expectations -discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator".

Does your paper address subitem 11a-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "They did not know if they were assigned to the intervention or control group and did not know which intervention was performed in the other group." Does your paper address CONSORT subitem 11b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "We performed a double-blind randomized controlled trial to assess the effectiveness of Hygie as a method for continuing education for GPs as compared with a traditional article reading activity with the same content." 12a) Statistical methods used to compare groups for primary and secondary outcomes 12a-i) Imputation techniques to deal with attrition / missing values Imputation techniques to deal with attrition / missing values: Not all participants will use the intervention/comparator as intended and attrition is typically high in ehealth trials. Specify how participants who did not use the application or dropped out from the trial were treated in the statistical analysis (a complete case analysis is strongly discouraged, and simple imputation techniques such as LOCF may also be problematic [4]).
Does your paper address subitem 12a-i? * Does your paper address subitem X26-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Participation was anonymous and voluntary. Participants began the study by clicking a link to teaching materials.

X26-iii) Safety and security procedures
Safety and security procedures, incl. privacy considerations, and any steps taken to reduce the likelihood or detection of harm (e.g., education and training, availability of a hotline) Does your paper address subitem X26-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : no harm to detect RESULTS 13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center 13b) For each group, losses and exclusions after randomisation, together with reasons Does your paper address CONSORT subitem 13a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "A total of 3398 GPs were invited to participate in this study by email. Of these, 269 participants (7.9%) accepted to participate in the study. Recruitment occurred between May 31, 2017, and June 27, 2017. A total of 108 participants completed the study and were analyzed." Does your paper address CONSORT subitem 13b? (NOTE: Preferably, this is shown in a CONSORT flow diagram) * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "The inclusion flow diagram according to Consolidated Standards of Reporting Trials recommendations [33] is shown in Figure  3." subitem not at all important 1 2 3 4 5 essential 14a) Dates defining the periods of recruitment and follow-up

13b-i) Attrition diagram
Strongly recommended: An attrition diagram (e.g., proportion of participants still logging in or using the intervention/comparator in each group plotted over time, similar to a survival curve) or other figures or tables demonstrating usage/dose/engagement.

Does your paper address subitem 13b-i?
Copy and paste relevant sections from the manuscript or cite the figure number if applicable (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "The inclusion flow diagram according to Consolidated Standards of Reporting Trials recommendations [33] is shown in Figure  3." Does your paper address CONSORT subitem 14a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

14a-i) Indicate if critical "secular events" fell into the study period
Indicate if critical "secular events" fell into the study period, e.g., significant changes in Internet resources available or "changes in computer hardware or Internet delivery resources" Does your paper address subitem 14a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : no secular events Does your paper address CONSORT subitem 14b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not applicable : not stopped early subitem not at all important 1 2 3 4 5 essential 16) For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups Does your paper address CONSORT subitem 15? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Baseline characteristics of participants in both groups were comparable (Table 1)."

15-i) Report demographics associated with digital divide issues
In ehealth trials it is particularly important to report demographics associated with digital divide issues, such as age, education, gender, social-economic status, computer/Internet/ehealth literacy of the participants, if known.
Does your paper address subitem 15-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Mean age was 40.9 years, there was a majority of women, and an urban setting was the most common" subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential 16-i) Report multiple "denominators" and provide definitions Report multiple "denominators" and provide definitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at specific pre-defined time points of interest (in absolute and relative numbers per group). Always clearly define "use" of the intervention.
Does your paper address subitem 16-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study A total of 108 participants completed the study and were analyzed

16-ii) Primary analysis should be intent-to-treat
Primary analysis should be intent-to-treat, secondary analyses could include comparing only "users", with the appropriate caveats that this is no longer a randomized sample (see 18-i).
Does your paper address subitem 16-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study not intent-to-treat "A total of 108 participants completed the study and were analyzed" 17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval) Does your paper address CONSORT subitem 17a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "The final SQ-20 mean score was similar in the 2 groups: Hygie group 4.9 (95% CI 4.6-5.2) and control group 4.6 (95% CI 4.2-4.9; P=.21, not significant). The final DQ-5 mean score (5-item post-test) was also similar in the 2 groups: Hygie group 5.0 (95% CI 4.6-5.4) and control group 4.7 (95% CI 4.2-5.1; P=.26, not significant). The mean individual change of DQ-5 score between pre-and post-test was significantly superior to 0 in the Hygie group with a mean gain of 1.6 (95% CI 1.2-2.1; P<.001) and in control group with a mean gain of 0.9 (95% CI 0.5-1.4; P<.001).
For the critical test of our trial, this mean individual change of DQ-5 score between pre-and post-test at 3 to 5 weeks was significantly superior in the Hygie group compared with the reading group, with a difference of 0.7 (95% CI 0.1-1.3; P=.02; Figure 4). For the question "In the course of your practice, did you use the knowledge that you learned through the teaching material?," the percentage of participants reporting "yes" was significantly greater in the Hygie group (77% in the Hygie group vs 53% in the reading group; odds ratio 2.9, 95% CI 1.2-7.4; Table 2)." subitem not at all important 1 2 3 4 5 essential 17b) For binary outcomes, presentation of both absolute and relative effect sizes is recommended 17a-i) Presentation of process outcomes such as metrics of use and intensity of use In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational definitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "average session length". These must be accompanied by a technical description how a metric like a "session" is defined (e.g., timeout after idle time) [1] (report under item 6a).