GPs were contacted via emails sent to medical networks in the Île-de-France region (including Paris) and by word of mouth, between March 25 and April 25, 2018. A reminder was sent 15 days after the first contact was made. For inclusion in the study, GPs had to be in training or practicing in primary care. The evaluation was voluntary and anonymous.

A web-based evaluation was carried in three steps. First, the participating GPs were asked to answer five sociodemographic questions. Their responses were rendered anonymous. Next, the GPs were asked to use the web interface for the use case “young adult woman with uncomplicated pyelonephritis.” The information displayed on the web interface was derived from a knowledge base describing 11 infectious diseases, 50 antibiotics, and 21 patient profiles, constructed based on French CPGs and clinical expertise (a description of this knowledge base is available in the literature [14]). Then, the GPs were asked to answer 17 questions relating to usefulness (3 questions), user-friendliness (3 questions), satisfaction (3 questions) and comprehension (8 questions). The responses were recorded on a 4-point Likert scale (ie, “absolutely disagree,” “tend to disagree,” “tend to agree,” and “absolutely agree”). At the end of the evaluation, GPs were provided with an opportunity to write optional free comments.

Given the opportunistic nature of the recruitment method used, it is difficult to estimate accurately how many GPs received the invitation email. The number of GPs contacted was estimated to range between 850 and 880.

In total, 117 GPs working within distinct GP surgeries accepted the invitation and assessed the web interface. More than half (73/117, 62.4%) were female. More than three-fourths (104/117, 89.0%) were under 40 years of age and had been in practice for less than 10 years. More than half (77/117, 65.8%) were working in private practice (Table 2).

Overall, 82.9% (97/117) of GPs found the display of drug properties useful (95% CI 76-90); 65.8% (77/117) reported that the interface improved their understanding of CPG recommendations (95% CI 57-74), and 59.8% (70/117) considered the interface useful for clinical practice (95% CI 51-69), as shown in Figure 3.

Only 54.7% (64/117) of all GPs were satisfied with the global interface (95% CI 46-64), probably because of dissatisfaction with some of the icons. Indeed, the majority of GPs appreciated the clinical summary (90/117, 76.9%) and filter (98/117, 83.8%) sections, whereas 48.7% (57/117) of them reported difficulties understanding some of the icons (Figure 3).

Five of the eight icons were considered sufficiently readable. The icons for recommended rank, ecological adverse effects, and taste were considered easily understandable by more than 80% of GPs; both the precious class and the side effects icons were considered comprehensible by more than 60% of all GPs. By contrast, three icons were not sufficiently readable; icons for antibacterial spectrum and convenient protocol were considered poorly understandable by nearly half the GPs and that for efficacy level was considered poorly understandable by about 84% of GPs (Figure 3).

Some GPs said that they found the web interface interesting but that it could take time for them to get familiar with it. They suggested several areas for improvement (Table 3).

For the clinical summary section, GPs suggested displaying more details about the probable causal microbes, patient history, treatment, and signs of gravity. For the filter section, they suggested adding filters for hepatic conditions and galenic forms. Regarding the recommended antibiotics section, the GPs thought it included too much information and needed to be simplified to improve readability. For example, some GPs said that only the most important properties should be displayed and that other properties should be hidden and accessible only on request (eg, nonprecious class, efficacy level, and taste). Surprisingly, some GPs reported that some properties were not important for the choice of antibiotic, whereas these properties were used by CPG experts in the formulation of recommendations (eg, ecological adverse effects, activity spectrum, antibiotic taste). Other GPs also had doubts about the nonprecious class property because they did not understand what this meant. Conversely, other GPs suggested including additional properties such as drug cost, drug-drug interactions, and contraindications. Some others said that some of the icons were not sufficiently intuitive and required revision. They suggested, for example, replacing the efficacy level by a graduated scale, speedometer, or weapon symbol. Finally, some GPs said that they thought the web interface could be useful for shared decision-making with patients.

The participating GPs were recruited via emails sent to medical networks and by word of mouth. This mode of recruitment was chosen because it is faster and cheaper than traditional approaches [27,28]. However, it may have introduced a selective participation bias [28] (eg, by selecting GPs keen on the use of new technologies). This bias was limited by using large mailing lists (>850 GPs), including GPs working in various health care centers, from a large French region (including Paris). The readability of the interface and its utility for improving GPs’ comprehension of CPGs were also evaluated. Assessments of readability and comprehensibility are part of the software lifecycle and may prevent serious usability problems, which often occur during the design of new technologies [29]. Early testing makes it possible to identify serious issues and to improve the interface before its implementation in a CDSS.

Other types of interfaces have been developed for comparing drugs in medical domains. With regard to Dopamine [30], for instance, double-entry tables were used to compare the contraindications and side effects of drugs. More sophisticated tables, including the possibilities of overlapping data and interactions, have also been developed, such as Twinlist [31] for medication reconciliation or rainbow boxes [32,33] for comparing drug information such as contraindications or side effects. Rainbow boxes were previously adapted for use in antibiotic prescription [34]; the resulting rainbow interface was perceived as easy to read for 27.5%-64% of GPs, depending on the clinical situation. However, as smartphones are increasingly being used by medical doctors for learning, information retrieval, and/or clinical decision support [35-38], it is important to consider interfaces more suitable for use with a smartphone. Tabular representations take up a large amount of space and are therefore more suitable for computer interfaces [39,40]. Here, an e-commerce interface adaptable for smartphones was designed as a means of overcoming this limitation. Graphs have also been used for drug comparisons, such as Rxplore [41] for the rapid review of potential drug events caused by drugs and Network graph [42] for chemotherapy treatment. However, in these cases, graphical representations were interesting because of the need to add quantitative information such as the results of clinical trials [42]. As antibiotic properties are qualitative data, graphical representations were not considered appropriate for this situation.

Other interfaces have been developed for the empiric prescription of antibiotics in primary care, for example, IAAP Smart phone [43], ARI Smart Form [44], and ABX TRIP [21] for acute respiratory infections. These interfaces display antibiotic recommendations without their justifications—that is, only the recommended antibiotics are displayed in a textual format without their properties. This type of presentation suffers from “the black box effect” [45] and may make GPs passive in their decision-making process, simply accepting the decision suggested by the CDSS. GPs may also be suspicious of the recommendations due to a lack of understanding of the suggestions made by the CDSS. In our proposed interface, drug properties were displayed to involve GPs in the decision process. This improves their understanding of the underlying reasons for recommendations and enables them to compare the advantages and disadvantages of the recommended antibiotics. Preserving GP autonomy is an important factor to be considered when trying to increase the chances of CDSS being successfully adopted [46].