We performed an open-label randomized controlled trial to assess the effectiveness of InsuOnline as a method for education of PCPs on insulin therapy for diabetes, as compared with a “traditional” onsite educational activity with the same content and same duration.

Subjects were eligible if they were medical doctors with an active register at a regional Council of Medicine in Brazil, were not specialists in Endocrinology or diabetes, were currently working at a public health care unit as a primary care physician (PCP), and were directly involved in the treatment of patients with diabetes in those facilities, with any degree of computer or gaming literacy.

Researchers contacted local public health authorities of some cities from the states of Paraná (Londrina and other cities from the 17th Health Regional, Curitiba and São José dos Pinhais) and Santa Catarina (Blumenau), who agreed to help with recruitment. Those authorities have enlisted PCPs in their area to attend a training session about diabetes, and then we contacted those PCPs via phone, mail, or email to invite them to participate in the study.

Those who agreed to participate were informed of the research procedures and were sent by email a link to an online informed consent form stored on Google Drive (Multimedia Appendix 1). Respondents were allocated to groups by simple randomization using an online random number generator [38] by the first author. Due to our expectancy of a higher attrition rate in the game group, allocation was made at a 2:1 ratio.

After filling the informed consent form, participants were randomized and informed about the group they were allocated to and received an email with a link to the baseline questionnaire (stored on Google Drive). Also, participants allocated to control group were sent an email with information about the time and location of their scheduled onsite learning activity. A printed questionnaire was applied to those participants of the onsite learning activity immediately after its ending.

Participants allocated to game group received an email with instructions on how to download and install the game, after filling the baseline questionnaire, and then were contacted (weekly, if needed) to check if they have finished the game. After that, they were sent the link to another Web-based questionnaire.

After 3 months, a link to the third questionnaire was sent to participants of both groups.

If the participants did not answer the questionnaire after a few days, reminders were sent at 3-7–day intervals (initially by email, then by text message, and finally by phone). If they did not answer after up to 6 consecutive reminders, they were considered as loss of follow-up.

Outcomes were self-assessed using written questionnaires which were applied to participants of both groups at 3 time points. The questionnaires were composed by different subscales at each time point: (1) at baseline, “competence” and “attitudes” subscales, plus demographic and professional data (for both groups); (2) immediately post intervention, the same “competence” and “attitudes” subscales for both groups, plus “game evaluation” subscale (for game group) or “onsite activity” subscale (for control group); (3) 3 months post intervention, “competence” and “attitudes” again, plus “importance for professional practice” subscale (for both groups).

The subscale of “competence” included 9 multiple-choice questions to assess factual knowledge about insulin and 11 clinical case vignettes with multiple-choice questions to assess problem-solving skills regarding insulin initiation and adjustment, which were created by our research team based on InsuOnline’s educational objectives (Textbox 1) and content. The subscale of “attitudes” included 9 Likert-type scale questions to assess attitudes and beliefs regarding diabetes and insulin, which were adapted from previous surveys [7,50].

The subscale of “game evaluation” had 16 Likert-type scale questions to assess playability, user satisfaction, and perceived educational utility of the game, with 1 additional free text item for additional comments from the participants, which were freely adapted from a questionnaire used to assess another game [51].

The subscale of “onsite activity evaluation” was composed by 10 Likert-type scale questions regarding methodology, user satisfaction, and perceived educational utility of the activity, and 1 free text item to collect additional comments, which were based on the ones from the “game evaluation” subscale, but adapted to an onsite CME session.

Finally, the subscale of “importance for professional practice” included 5 Likert-type scale questions and 2 free text items to assess participants’ opinion on the actual impact of the educational activity (game or onsite learning session) on their current professional practice, which were also created by authors.

All subscales were previously tested and refined in our pilot study with medical students and residents [36,37].

An additional subscale with 5 multiple-choice questions and 7 free text items was applied only to participants allocated to the game group who did not finish the game 60 days after receiving instructions for download or who did not respond to post-intervention questionnaires, to assess reasons why they were unable to play the game, and their perceptions about it.

All instruments used in this study are available in Multimedia Appendix 2.

Applicability of each intervention was defined as the proportion of participants allocated to each group that have received the intervention and did finish all tasks required in each intervention.

The primary outcome of this study was the score in the competence subscale in the immediately post-intervention self-assessment. Secondary outcomes were: score in competence subscale 3 months post-intervention; participants’ attitudes regarding diabetes and insulin post-intervention; participants’ acceptance and satisfaction with learning activities (game or onsite CME session); and applicability of the intervention (game or onsite CME session).

Only subjects who filled the baseline and the immediate post-intervention questionnaires were included in the analysis. Data from questionnaires were downloaded from Google Drive online forms or typed from printed questionnaires to worksheets on Excel (Microsoft Corporation) and then analyzed on Epi-Info 7 (Centers for Disease Control, Atlanta, Georgia) for descriptive analysis. Demographic data were compared among groups at baseline using chi-square test for proportions or Student t or Kruskal-Wallis test for continuous variables, as suited. Correlation among baseline scores on the competence subscale and demographic baseline data was assessed using Pearson r or Student t test on SPSS 14.0 (SPSS Inc). Comparison of the scores on competence subscale by group and by time point was performed by repeated-measures analysis of variance (ANOVA), with the Tukey post hoc test when a significant difference was detected, and by simple comparison between 2 groups at each time point using Student t test, on the statistical software R (The R Foundation). Proportion of subjects with scores of 90% or more on competence subscale and the frequency of “agree” answers on the Likert-type scale items for assessing attitudes were compared by group and by time point using chi-square test (followed by post hoc pairwise comparisons using Bonferroni correction when significant). Absolute increase in competence score from baseline to immediately post intervention was estimated on Epi-Info 7. The effect size of the group variable on the competence score was estimated using Cohen d test on R software. Reliability (internal consistency) of the subscales used in this study was evaluated using Cronbach alpha. Comments made by subjects in the free text items were reviewed by content analysis. Statistical significance was defined as P<.05.

Participation was anonymous and voluntary, and all subjects filled an informed consent form (Multimedia Appendix 1), in accordance with Brazilian Health Ministry’s regulations for research on human beings. The study protocol was previously approved by Londrina State University Research Ethics Committee (UEL, #051/2011 and #051/2012), and by all local public health authorities from the cities where the study was performed.

A total of 257 primary care physicians were contacted and assessed for eligibility during the recruitment period (July 2014 in Curitiba and São José dos Pinhais, Paraná; August 2014 in the cities from the 17th Health Regional of Paraná State except Londrina; September to October 2014 in Londrina, Paraná; and October 2014 in Blumenau, Santa Catarina); 170 were randomized, and 134 were included in the final analysis (however, 4 subjects from the game group and 7 from the control did not answer the third questionnaire, 3 months post-intervention). The CONSORT diagram [52] for participant flow is shown in Figure 3.

Baseline characteristics of subjects in both intervention groups were comparable, with no significant differences (Table 1).

Our results have shown that the game InsuOnline, used in real-world conditions (in players’ own computers and in their own flexible time schedule) was highly effective for education of PCPs on insulin therapy for diabetes, compared with an onsite CME activity with the same content and same approximate duration. In fact, competence improvement was better with the game than with the onsite learning session. InsuOnline was also applicable, with about 80% of the subjects with the more varied degrees of computer or gaming literacy being able to finish it with little or no external help. Both interventions were very well-rated by participants, regarding engagement, realism, and perceived educational value. The good evaluation of the onsite CME session proves that InsuOnline was compared with a valid control activity (a gold-standard active comparator). Three months after both the interventions, doctors from both the groups said that they were feeling more secure and more prepared to help real patients with diabetes in their daily professional practice, with even better results in the game group.

Games and simulators are being increasingly used for education of health care professionals on various topics [55-57], but, to our knowledge, this is the first report on the educational effectiveness of an electronic game for education of medical doctors about insulin therapy for diabetes.

Games are a promising way to deliver CME, for many reasons. One reason is their educational adequacy: well-designed games incorporate all the main principles of Adult Education, such as individualized, self-pacing, contextualized learning, with active experimentation and appreciation of previous knowledge [15,16,40-43]. The inclusion of game elements adds entertainment to the learning experience, increasing learners’ motivation to practice and to learn, which renders the learning experience more enjoyable, more engaging, and potentially more effective [40,58]. Games can be designed to simulate an infinite number of medical problems, and are easily updated as needed [15,40]. Finally, games are more flexible than traditional CME, respecting learners’ own time availability and learning pace, and are much more scalable [23]. So, we believe that well-designed and validated games will soon become the gold standard option for delivering large-scale CME.

Validation of games for health professional education is a recent and growing research area. Most available games for health professionals’ education have not passed through an objective validation process, and many of the published studies on the field are biased [59]. So, there is a great need of good quality studies which can provide solid evidence to support the fast-growing science of games for health. Our randomized-controlled trial was designed to contribute to this body of evidence by following the guidelines for research on health games effectiveness as proposed by Kato, who suggested that studies in this area should attempt to apply the same scientific rigor typical of health sciences [60].

As there is a variety of proposed methods for validating games for health, we opted to use multiple methods for validation of InsuOnline. One of the simplest validation approaches was proposed by Olsen [61], and includes only 3 aspects: usability, playability, and educational effectiveness. The first two aspects were very well rated for InsuOnline, as previously reported [36], and our current results support the third one. However, the best method to validate a game that was designed for CME is the approach to validation of CME activities described by Moore [62], that proposes a continuum of 7 levels: (1) applicability (user participation), (2) user satisfaction, (3) gain of factual and procedural knowledge (learning), (4) gain of competence, (5) improvement of learner’s performance, (6) improvement of patient health, and (7) improvement of community health. InsuOnline has already met the first 4 criteria. The following 3 steps (change in PCP’s professional performance and improvement of their patients’ glycemic control, and, hopefully, improvement of populational risk of DM complications) will be the focus of future research.

Attitudes and beliefs of PCPs in our sample were similar, at the baseline, to the ones of family doctors in United States [7] and in Arab-speaking countries in the Middle East [50], although the Brazilian doctors who participated in our study seem to be more often inclined to perceive the initiation of insulin therapy and the training of the patient with DM for using insulin as difficult and complicated tasks (78% and 54% of our subjects, respectively, compared with 66% and 42% in United States, and 48% and 25% in Middle East), which underscores the urgent need for effective educational interventions.

Reasons for the better results observed with the game than with the onsite learning activity, in this study, can be related to characteristics of the games that foster learning and attitude (and behavioral) changes [63]:

1. The inclusion of behavioral change procedures into the game, such as: goal setting, decision-making, problem solving, goals re-evaluation, social rewards; and

2. The use of a story (the game plot), and the inclusion of behavioral change concepts into that story. Players’ identification and empathy for the story protagonist may contribute to change players’ attitudes, reflecting the attitudinal change that occurs with the protagonist character in the game, in a process known as “modeling” [63].

Also, the higher emotional involvement of the learner with a game than with a traditional CME activity is another likely factor, as affection is a powerful determinant for changing attitudes and behavior [64]. Intrinsic motivation, a very effective learning catalyst, probably is one of the biggest advantages of games in relation to more traditional modalities of education [65]. Our game was designed to produce a high level of players’ engagement, by careful disposition of different game elements (realism, identification with context and characters, increasing-difficulty challenges, humor, clear objectives, immediate and intense feedback for each players’ action, progress monitoring, and rewards). With that, the majority of participants stated that the game was “fun” and that it captured their attention all the time.

Some limitations of our study should be pointed out. First, blinding of the participants (and researchers) was not feasible due to the nature of the interventions, and that may have affected some of our results, especially the subscales of evaluation of the activities, as many subjects may have found it exciting or unusual to play an electronic game to learn about a medical topic. However, we think that our primary outcome (competence for prescribing insulin) was not compromised, as it was objectively assessed by standardized multiple-choice questions.

Second, the instruments used in this study were not previously validated, because they were developed by our team to assess the specific outcomes we addressed in this study. The use of customized instruments is strongly recommended by Moreno-Ger [66], who argues that generic questionnaires are usually not useful for assessing games that can be very different in their objectives, target audiences, and needs. At our favor, we can say that our instruments were extensively reviewed by our team of experts in endocrinology and medical education, which warrants their face and content validity, and our subscales have shown acceptable reliability, as measured by their Cronbach alpha in the range of .7 to .8, with an exception of the attitudes subscale, which had very poor internal consistency [54]. The authors decided to maintain this subscale because it had questions similar to the ones used in previous surveys [7,50], allowing their comparison with our results.

In third place, this study shows only intermediate outcomes (PCPs’ knowledge, skills, attitudes, competence regarding insulin initiation, and adjustment for diabetes), but we don’t know yet if the game will have any impact on actual health professionals’ performance or on their patients’ measures or outcomes (“hard” outcomes), which we expect to assess in future studies.

Finally, although PCPs’ competence for prescribing insulin is essential, it is very likely that a simple intervention on the education of medical doctors may be insufficient to induce a significant change in the process of care delivered to patients with DM in primary care. A multifaceted intervention, aiming at several aspects of DM management, should probably be more effective to improve care and control of DM on a primary health care level, but it would demand conjugated efforts from government and society to improve issues such as availability of multiprofessional teams, availability of better insulins and oral drugs, availability of reagent strips for self-monitoring of blood glucose, better access to diabetes education, and others.

We think our results can be generalized for most primary care physicians worldwide, as our sample included medical doctors from different geographic areas, different age groups, different levels of specialization and experience, and any degree of computer or gaming literacy. Also, our game was tested in what we believe to be the “real-world conditions” for the use of an electronic game for distance CME: in players’ own computers (with a wide range of possible equipment configurations), with very little or no external help from our team, and in their own spare time. Our results show that InsuOnline is a flexible, applicable and scalable option for large-scale CME on diabetes.

In this “real-world conditions” randomized controlled trial, InsuOnline, the first electronic serious game designed for medical education on insulin therapy for diabetes, was applicable, very well accepted, and highly effective for education, with even better results for improving primary care physicians’ competence and attitudes related to diabetes and insulin than a gold-standard onsite CME activity with the same content and duration. For this reason, we believe InsuOnline is a valid tool for large-scale CME on DM, with advantages manifested in its easy Web dissemination, customizable content, and accordance with Adult Learning principles. We hope it can contribute to improve PCPs’ performance and optimize the quality of care offered to patients with DM in primary care, eventually improving patients’ glycemic control and reducing the risk of DM complications.