Antibiotic resistance is a major public health problem worldwide that is mostly fostered by inappropriate use of antibiotic medications. At the same time, data and advances in health care are growing not only in quantity but also in complexity; thus, health systems, practitioners, and even patients are required to be in a constant learning state to achieve effective monitoring and evaluation [1].

With the near total ubiquity of mobile technologies, mobile health (mHealth) is becoming an increasingly established field with important results in different domains [2-4]. This gives rise to new possibilities in enhancing clinical decisions in all medical fields, including antibiotic prescription and dispensing.

In this framework, the project eHealthResp proposes to create and evaluate eHealth tools to support clinical decisions and patient empowerment in the management of upper respiratory infections. These tools include an web-based course targeted toward physicians and pharmacists and a clinical decision support system (CDSS) mobile app targeted toward physicians, pharmacists, and patients.

At a later stage of the research, both tools will be used by participants of an educational intervention about antibiotic prescription and dispensing for upper respiratory symptoms supported by the eHealthResp project.

Expected outcomes include a decrease and improvement in the use of antibiotics as well as a comprehensive list of guidelines in designing and implementing feasible and usable tools for CDSS in a broad scope, particularly for cases involving the upper respiratory system.

In the realm of these predictions, the work in progress and the main issue of this paper focus on the methodology and subsequent results within the interface design stage of the mobile app tool for smartphones running Android or iOS operative systems.

This research stage can be summarized within the following 4 components:

Currently targeted toward Android and iOS mobile devices, the eHealthResp mobile app is under development and consists of a diagnostic tool designed to enhance the clinical decision set facing upper respiratory symptoms. The algorithm behind the app and a version of the app were previously validated and designed for the pharmaceutical scope by a research group from the University of Santiago de Compostela based on the work of Molero et al [7]. This served as the basis for the development of the new proposal.

At the end-user level, the tool consists of a wizard that guides the user through the dichotomous key-like algorithm and, in cases in which a disease is identified, presents the end results in a diagnosis format, including know more, treatment, prognosis and when to derive information. The workflow was kept linear, straightforward, and with little to no deviations from the main course to keep the user’s focus on the diagnosis.

Although the previously developed app, available on the Google Play store, presents the wizard at a functional level, the interface design challenge was not considered when it was developed. At the design level, the major weaknesses of the app concern the use of stretched pictograms, misuse of Android navigation elements (eg, tabs, buttons), lack of consistency between pages and graphical elements (eg, buttons, pictograms, backgrounds), unoptimized image compression, and an overall assessment reflecting an unpleasant interface that fails to entice the user to trust and reuse the tool [8,9].

Using the Scopus database, a combination of the keywords mobile app (or mobile health or app), clinical decision, and respiratory was used to search for similar studies.

Due to the lack of direct references concerning diagnosis apps and design for mHealth, complementary searches were held combining the terms algorithm, mobile, design, diagnosis, or diagnose, and respiratory. From this search, 9 other articles were selected based on title and keywords. In total, 47 articles were selected and analyzed.

In addition to this review, a nonexhaustive benchmark-like search was conducted to gather a glimpse of the state of art in interface and design options for apps in the respiratory and CDSS fields. For this, separate searches were held on the tags respiratory and clinical decision in the 2 most relevant app stores (Google Play and App Store). For both the tags and stores, a subsearch for paid apps was performed.

The inclusion criteria were as follows: (1) respiratory system representations, (2) clinical decision tools, (3) quiz formats, and (4) informational or educational content involving the respiratory system. Apps were listed if they included at least one of the previous criteria. Apps that had cross-functionalities with the app under development were highlighted.

For each app, the list included inherent details (title, icon, link, creator, classification, price, description, last update, size, number of downloads, and screenshots) and 2 evaluation lists (pro- and counter-considerations) of the interface characteristics concerning the given screenshots and listed functionalities and based on the literature review. When available for free, highlighted apps were downloaded for a deeper analysis.

To achieve an end design that suits the app users (physicians and pharmacists), the interdisciplinary team involved in this project worked together toward the development of the alpha version of the interface (Figure 2). In addition to periodic meetings, a questionnaire was used to assess the opinions of experts in pharmacology and pharmacoepidemiology about different interface solutions, assigning experience-based values throughout a participatory strategy [10]. This includes understanding the needs and preferences of experts and their peers (physicians and pharmacists) regarding content (script and imagery) and visualization formats. Using fast prototyping tools (Sketch and Marvel App), high-fidelity interface mock-ups were made according to participants’ previous feedback and presented to the participants.

To limit the time spent in developing high-fidelity mock-ups, the design covered strictly necessary paths to walkthrough the most decisive type-pages. Redundant, secondary, and broadly studied representations were avoided, such as pages with the same structure but different content and the Settings page.

To attain an outsider point-of-view and opinion, a first series of peer review sessions was held by a group of 5 experts in interface design. This group was conveniently gathered based on the availability and diversity of the members’ academic and professional backgrounds.

After an introduction to the research framework and approach, each expert reviewer received a PDF document with the interface mock-up pages and an interactive high-fidelity mock-up version of the interface in the web-based app Marvel App. Both the PDF document and the Marvel mock-up presented 2 walkthrough paths: one for a light version of the interface, that is, brighter, and the other in a darker mode. To assess the quality of the navigation and graphical design of the interface, a ready-to-fill spreadsheet was provided. This document, an adapted version of the web-assessment tools, 38 page layout and visual design usability guidelines and 29 navigation and IA usability guidelines (IA stands for information architecture), both part of the “247 web usability guidelines” by David Travis [11]. The guidelines provided by Travis for web usability consist of bullet point lists that are meant to be used as assessment tools to improve consistency and design practices. As the author emphasizes, these guidelines could and should be adapted to the specific context in scope. The assessment tool was translated from English to Portuguese and adapted to keep the focus on the mobile app scope and the 2 main dimensions concerning interface and usability: (1) layout and visual design and (2) navigation and tasks. The adapted version presents a set of 30 parameters to assess layout and visual design and 21 parameters for navigation and tasks. Each parameter presented a statement suggesting a good assessment for a specific criterion (eg, “1. Content density is appropriate for target users and relevant tasks”). The reviewer had the possibility to classify each parameter through an adapted version of the Likert scale, measuring from 1 (Do not agree) to 4 (Totally agree), so that the design team could perceive the importance of the parameters in a more hierarchical way and determine whether to ignore or pay attention to each parameter. The choice for the lack of a fifth neutral middle value was taken to force the reviewer’s positioning, as applied in the original binary (complies or does not comply) guideline tool by Travis. Similar to the original tool, each parameter had an open-ended space for observations.

The mean and median were calculated to interpret the quantitative data collected throughout the reviewers’ evaluation using the adapted Likert scale for each bullet point. For the qualitative information gathered through the commentaries, a mean median was calculated to express the number of commentaries by bullet point. Given the relatively small sample of reviewers, every commentary was taken into consideration regarding the specific layout elements that were being referred to and the importance of the issue being addressed.

The search was conducted between February and April 2019, and a total of 297 references were found. Only 34 articles directly addressed or focused on the searched issues. These 34 articles comprised a publishing period between 2004 and 2019. In total, 9 of the articles found describe studies conducted by the same organization—Allergic Rhinitis and its Impact on Asthma (ARIA) [12-20].

Some of the studies directly addressed the research framework (mobile interface design for CDSS in respiratory conditions); the majority of these focused on the effectiveness of a particular app in enhancing diagnosis and prescription, but few of them consider the user experience with the interface and the hedonic qualities of the interface that influence its long-term use [21]. Nonetheless, they gave important hints for the production of valuable and understandable tools.

The following list presents the main results of the overview, with the articles found and 13 other references that were derived from the original 34:

The state-of-the-art search covered a period between March 20 and 30, 2019, and produced a list of 98 apps (34 on the App store and 64 on the Play Store. In total, 15 of these apps, were highlighted and 1 was found in both stores).

Overall, the analyzed apps exhibited the following identified characteristics:

As already mentioned, this stage produced a new interface proposal featuring one static and one interactive mock-up. This design took into account the guidelines found during the literature search and state-of-the-art survey stage, the CDSS tool requirements expressed through the previously developed app (eRes), and the researchers’ considerations. Among the considerations, the practitioners suggested the design of 2 themed versions (light and dark), for the purposes of personalization and comparison of the visibility of the layout structure. Each version consists mostly of a color switch between the content page and the background color. The secondary or action color was maintained, and the problem or error color was adjusted to improve its visibility.

Further considering the color issue, different chromatic choices were used to visually distinguish the interfaces for different users (physicians, pharmacists, and patients).

To convey the project aesthetic and credibility standards, the original interface layout was broadly redesigned, including reconsideration of the interface elements and their positioning. This includes not only the color aspect but also the typeface, buttons, animations, and redesign of the illustrative pictograms. Small tweaks to the navigation structure were performed on the Results page to gather related information, highlight important content, and hide secondary information in secondary pages.

The input given was used to produce the beta version and included the reconsideration of gradients (to be avoided), layout (repositioning elements such as questions and answers and buttons such as Repeat test), pictograms (redesigned and avoided in some cases), typefaces (avoiding bold and light versions), and the design of a confirmation page before finishing the test.

After the redesign, the new interface was presented in person through meetings with 2 of the reviewers, who were selected based on the amount of input given previously. These meetings were also intended to ensure that the design team correctly interpreted their responses and assessed whether the new proposals effectively rectified the issues that were found.

With 4 as the maximum value of the abovementioned Likert scale, the interface scored a mean of 3.16 (SD 0.45; median of the means 3.2) for layout and visual design and a mean of 3.43 (SD 0.33; median of the means 3.51) for navigation and tasks (the data are presented in detail in Tables 1 and 2). Within the chosen criteria, these results can be acknowledged as confident positive evaluations of the proposed interface.

Nonetheless, some parameters scored moderately high values: 22. Clickable items and content (buttons) include redundant labels/subtitles for layout and visual design was the lowest scored item, with a mean evaluation score of 1.8 (median 1, SD 1.1). Other low-score parameters include 11. Colors are used to structure and group items (mean 2.6, median 2, SD 0.89) and 16. Attention-grabbing elements (eg, animations, bold colors, different sizes) are used with caution and only when necessary (mean 2.4, median 2, SD 0.55).

The lowest score for the navigation and tasks scope concerned the parameter 17. The main sections of the app are available from any screen and there are no dead ends (mean 2.6, median 2, SD 0.89). The following lowest value already achieved an appreciably positive value of 3/3 for 8. The app correctly anticipates the user’s next intentions.

Although the quantitative approach provides important hints on what to look for and an overall assessment, the open-ended commentaries enable better understanding of the reviewer’s concerns. In this section, layout and visual design received a mean of 1.2 comments per parameter (median 1); the abovementioned parameter 16 was the one that received the most feedback, with comments from 4 of the 5 peer reviewers. All comments for this parameter warned about different issues such as text sizes, contrasts, visual weight, and sparse use of colors. The parameter 11. Colors are used to structure and group items had 3 comments, all of which highlighted the absence of a more variable color palette (eg, “Color variability is not something that goes into this app”). In addition, with 3 comments, the parameter 20. Radio buttons and check boxes are used appropriately raised issues concerning the interaction limitations of the mockup.

The navigation and tasks section received approximately 0.714 comments per parameter (median 1). The most frequently commented parameter (3 comments) was 17. The main sections of the app are available from any screen and there are no dead ends. The comments reflected on the one-way-path aspect of the app, the limitation of the mock-up in turning back one action, the lack of a submission confirmation page (“I would say that before presenting the likely diagnosis there should be an confirmation of intention to ‘submit’”) and the difficulty in locating the button to go back to the homepage and repeat the test (“I can’t find a way to go back to something that resembles the homepage”).

In this study, we present the main stages and outcomes of the current developments in the interface design for the eHealthResp mobile app as a means to produce guidelines for mobile interface development for other CDSS tools with similar characteristics. Guidelines such as these, though available, are still scarce and lack validation [60].

The literature review highlighted the possibility and need for mHealth solutions to enhance diagnosis [13,26-28,31,33,35,40,42,50,52,60]. It gave a glimpse of the large number of mobile apps currently available within the health topic [43-45] and also flagged the lack of cohesive evaluation standards among them [25,26,28,38,41,60]. However, it helped collect important guidelines to foster better CDSS and mHealth solutions [13,15,18,19,27,30,36,44,46-52,61] to positively impact the quality of care regarding diagnosis [47] and potentially support overburdened medical education programs, promoting better patient care [30] and better, quicker, and more confident clinical decision processes by physicians [35].

As one of the most prominent studies found in the literature, Mobile Airways Sentinel networK (MASK), part of the ARIA initiative, focuses on the design and implementation of tools and guidelines for tool development in the scope [13,15,16,18-20,61].

Within several references to the ARIA project, Courbis et al [19] described a cascade-like methodology for implementing clinical decision support from paper guidelines to the MASK mobile app. A similar approach was adopted in building the eHealthResp app, including collaborative ways of designing and evaluating the solution and transforming the validated algorithms into a user-friendly interface.

The eHealthResp mobile interface design also follows a very similar design methodology to that adopted in mPneumonia [46]. The study makes use of prevalidated algorithms transforming them from paper into a step-by-step, user-friendly assessment questionnaire for mobile interfaces. The team also focused on the feasibility and usability, and unlike our study, they managed to gather acceptability levels. Most of the problems raised in the mPneuomonia project were conveniently approached while designing the first version of the eHealthResp mobile interface and were not raised during the appraise by the peer reviewers.

In accordance with the literature review, the state of the art review reiterates the existence of many apps available within the health care spectrum [43-45]. Despite this, these apps are presented through poor classification and evaluation systems [28], making it difficult to search for trustworthy and easy-to-use apps for a specific issue such as CDSS for upper respiratory infections.

Alongside conclusions by Panesar et al [50], we believe that a well-designed and accepted smartphone app can increase awareness of the importance of antimicrobial stewardship and influence some prescribing behaviors. The right information in the right context can reduce uncertainty, particularly in the antibiotic prescription realm [32].

Within the process of designing the interface, the most important factor was the design team’s interdisciplinary dimension. We tried to convey the warnings from researchers such as Litvin [49] and Rawson [51] about the need to predict the tool inclusion in the clinical workplace and grant the perception of usefulness to assist in decision making. This was done by directly involving the clinical scope throughout the design study from an early stage. This involvement was granted by an interdisciplinary design team with experienced members in the clinical field and also by targeting the app toward the pharmacists and physicians’ participants of the seminar (introduced in the Background section of this paper).

Despite confirming the beneficial outcomes for patients, Terry [28] flags the ongoing issue of classifying and rating mobile apps for health and calls for the inclusion of physicians, patients, and caregivers in the evaluation process. In addition, while detecting strengths, weaknesses, opportunities, and threats of smartphone-supported diagnosis for the particular case of allergy diseases, Pereira [26] highlights the lack of validation for this type of tool (for diagnostic decisions) and calls for multidisciplinary studies, similar to the research in focus on this paper, to obtain high-quality and useful tools.

These conclusions go along with the outcomes expressed throughout the design process, during which insightful outcomes arose from several meetings and questionnaires regarding preferences and worries predicted by the team members closer to the end-user’s community. Although some authors call for automated evaluation tools [36], we retained the use of questionnaires and meetings with experts to evaluate the designed app as a pragmatic way of making quick assessments and fostering the interaction between the design and pre-evaluation processes.

In addition, and because the contexts of use and users were already clearly predefined and represented among the research team members, it was possible to address the issues with a close to contextual design approach without the need for deep or direct research within the broad and complex scope of the study and all the limitations that this approach could imply [25].

At the same time, fast, interactive, and high-fidelity prototyping was revealed to be a major key factor for the co-design approach. This allowed other nonexpert designers to understand the approach almost seamlessly without requiring any kind of abstraction or written descriptions. Problems arose only with specific limitations of the prototyping tools, such as the inability to conveniently represent horizontal scroll, specific content animations (eg, icons mutations, element dislocations), and different screen aspect ratios.

Other interface design insights were taken into consideration; for example, Martínez-Pérez et al [44] highlight the need to avoid the use of text-only interfaces, making use of the interactivity, images, and logical decision trees throughout algorithms in a step-by-step approach to input data and restrict the input need to the minimum, reducing the time required to complete the diagnosis.

Guidelines for Android interfaces [53] were used to convey consistency with the primary system (Android) in which the mobile app will run. After the app is fully developed, convenient adaptations will be made so that the interface can be used equally in the iOS ecosystem [62].

In the scope of web design, Lindgaard [54] hints at the importance of immediately perceived esthetics, beauty, and visual appeal to grant hedonic values and urge the user to trust and use the tool. However, in the realm of web design, we believe that these values can be adopted in any design project that relies on the visual sense to obtain the user’s trust.

In a study by Shneiderman, the “golden rules of interface design” [59] were addressed throughout research by Gong and Tarasewich [55] to re-adapt these rules into “guidelines for handheld mobile device interface design.” These guidelines comprise original, adapted, and new guidelines that enable frequent users to use shortcuts; offer informative feedback; design dialog to yield closure; support internal locus of control, consistency, reversal of actions, error prevention and simple error handling; reduce short-term memory load; design for multiple and dynamic contexts; design for small devices; design for limited and split attention; design for speed and recovery; design for top-down interaction; allow for personalization; and design for enjoyment.

Nielsen and Budiu [58] highlight the general characteristics of mobile human-computer interactions. Among other things, they underline the importance of a clear start-up screen and the consistency between app pages and branding.

The authors also bridged the design stage to the evaluation stage, often linking these stages in a circular manner that fosters redesigning and re-evaluation. Authors such as Kushniruk et al [48], Shneiderman et al [59], authors from the Interaction Design Foundation [56], Nayebi et al [57], Nielsen and Budiu [58], and Travis [11] stress the importance of good usability testing, qualitative research, and methodologies, including usability heuristics with well-tested design principles for inspection, walkthroughs, action research methods, and concepts such as affordance.

Despite all the evidence that mobile devices are valuable tools for clinical decision-making by both physicians and pharmacists, there is still a need for rigorous evaluation, validation, and best practices for development to ensure the end-quality and safety of the tool [33]. Although the literature was very positive regarding the use of mobile technologies, it also warned about user anxiety issues, limited access to technologies for some, and security concerns [37].

A peer review was necessary to understand the interface limitations in a broad scope. The adopted methods and tools were revealed to be useful and suitable to quickly assess the reviewers’ opinions about the interface issues. Overall, the 5 reviewers provided approximately 0.96 comments per parameter, reflecting considerable efficiency of the chosen parameters in raising potential problems. As the main research goal was to detect potential issues rather than to conduct an overall evaluation, the adaptation of the guidelines by Travis [11] enabled a quick and efficient evaluation setup. The shortening of the classic 5-point Likert scale to 4 points allowed a relatively short sample of peer reviewers to express their opinions in a more binary (positive or negative) way for each guideline. The downside of this new scale concerns the negative assessment of some bullet points that could otherwise be classified as neutral evaluation and the overlooking of some positive points. Despite this concern, the reduced number of reviewers and the need to interpret every comment made allowed us to carefully assess each bullet point.

The results supported the creation of the beta version of the interface (Figure 3), addressing most of the detected problems. The version was further reassessed by 2 of the previous reviewers, validating the new design. In this second stage of validation, performed by 2 of the more critical reviewers, the interface was classified as clear and easily usable.

Overall, the processes of co-design, evaluation, redesign, and re-evaluation produced valuable outcomes, addressing major problems of the initial design and proving the processes to be an efficient strategy to speed up the design process. As the main outcome, the study enabled us to create a guidebook for the development, with recommendations (Figure 4) explaining the rationale behind the design choices and the constructive rules for the interface. This guidebook allows developers to have a clear perception of the composition of the interface, summarizing the following chapters: (1) introduction, (2) layout basic elements, (3) page types, (4) specific layout behavior, (5) grid or relative distances, (6) themes, (7) color scheme, (8) typeface, (9) list of pictograms, (10) transitions between pages, and (11) animations.

The major limitation of the study is the relatively small design team. Although the small team size contributed to speeding up the decision processes, including only 2 HCI experts and 3 experts in pharmacology and pharmacoepidemiology could preclude the representation of the variety of possible contexts of use.

As already highlighted in the previous sections, limitations within the interactive mock-up also presented an issue to the nondesigner team members and reviewers’ assessment in independently interpreting the solution. Specific limitations of the prototyping tools in representing items such as horizontal scrolling, specific content animations (eg, icons mutations, element dislocations), and different screen aspect ratios, caused some confusion in interpreting the solution. In addition, the fact that some buttons being represented were not interactive and led to no alternative path caused some uncertainty regarding the true meaning of these elements among the reviewers and nondesigners.

These limitations could only be clarified in person and within the second evaluation stage with the reviewers.

Future work should include the implementation of the app and its confrontation with established assessment criteria such as mHealth evidence reporting and assessment [63].

After this step, usability tests with end-user input must be performed. Conveniently selected physicians and pharmacists in their workplaces will assess the usability of the mobile app using the System Usability Scale as a tool. Their feedback and adherence will better ascertain the effectiveness of all solutions [41].

After the design is revised and established, the research will undertake a pilot study to evaluate the effectiveness of the aforementioned tools, covering 20 primary care physicians, 20 community pharmacists, and 50 patients selected by key informants. This study will gather quantity and quality indicators as response variables within the context of antibiotic consumption to be statistically analyzed on an intention-to-treat basis. The tool should undertake a validation, in which the outcomes from the use of the mobile tool will be compared with those within the usual clinical decision setting (without the tool) together with a user survey regarding the user experience and usefulness of the app [23].

Understanding the true impact of mHealth tools is still an uncertain task, as Forrest et al [39] concluded from their analysis of CDSS solutions. At the time of the study, it was still difficult to perceive the true impact of these tools on patient health outcomes. Although these solutions can significantly improve adherence to antibiotic prescription guidelines, providing easy access to these tools may not be sufficient to achieve higher levels of adherence [34].

As deepened in the literature and state of the art reviews, there is a need to develop recognizable standards in the development of mHealth solutions for CDSS in upper respiratory symptoms control. For this, the main factors influencing the success of these tools must be identified to complement the existing guidelines for mobile development [13,42,44,50-55]. More specifically, there is a need to define requirements relating to layout and content design for usability, acceptability, and usefulness of the app contents and features.

In an attempt to answer the question “How do we develop an interface for an app to support physicians, pharmacists, and patients to properly use previously validated algorithms for upper respiratory symptoms?” the research performed to date has contributed to clarifying some relevant aspects. The use of state-of-the-art tools for high-fidelity prototyping can be crucial to speed up the design process for a multidisciplinary team, not only because it can clearly represent the designers’ conventions to the rest of the team but also because it can help to easily and quickly integrate ongoing suggestions, allowing for a highly interactive co-design process that conveys the team’s concerns.

The developmental methodology enabled us to produce a set of guidelines or templates to produce an app that conveys the requirements of the app and aids the implementation stage.