Background: The low levels of control of hypertension and diabetes mellitus are a challenge that requires innovative strategies to surpass barriers of low sources, distance, and quality of health care.
Objective: The aim of this study is to develop a clinical decision support system (CDSS) for diabetes and hypertension management in primary care, to implement it in a resource-constrained region, and to evaluate its usability and health care practitioner satisfaction.
Methods: This mixed methods study is a substudy of HealthRise Brazil Project, a multinational study designed to implement pilot programs to improve screening, diagnosis, management, and control of hypertension and diabetes among underserved communities. Following the identification of gaps in usual care, a team of clinicians established the software functional requirements. Recommendations from evidence-based guidelines were reviewed and organized into a decision algorithm, which bases the CDSS reminders and suggestions. Following pretesting and expert panel assessment, pilot testing was conducted in a quasi-experimental study, which included 34 primary care units of 10 municipalities in a resource-constrained area in Brazil. A Likert-scale questionnaire evaluating perceived feasibility, usability, and utility of the application and professionals’ satisfaction was applied after 6 months. In the end-line assessment, 2 focus groups with primary care physicians and nurses were performed.
Results: A total of 159 reminders and suggestions were created and implemented for the CDSS. At the 6-month assessment, there were 1939 patients registered in the application database and 2160 consultations were performed by primary care teams. Of the 96 health care professionals who were invited for the usability assessment, 26% (25/96) were physicians, 46% (44/96) were nurses, and 28% (27/96) were other health professionals. The questionnaire included 24 items on impressions of feasibility, usability, utility, and satisfaction, and presented global Cronbach α of .93. As for feasibility, all professionals agreed (median scores of 4 or 5) that the application could be used in primary care settings and it could be easily incorporated in work routines, but physicians claimed that the application might have caused significant delays in daily routines. As for usability, overall evaluation was good and it was claimed that the application was easy to understand and use. All professionals agreed that the application was useful (score 4 or 5) to promote prevention, assist treatment, and might improve patient care, and they were overall satisfied with the application (median scores between 4 and 5). In the end-line assessment, there were 4211 patients (94.82% [3993/4211] with hypertension and 24.41% [1028/4211] with diabetes) registered in the application’s database and 7960 consultations were performed by primary health care teams. The 17 participants of the focus groups were consistent to affirm they were very satisfied with the CDSS.
Conclusions: The CDSS was applicable in the context of primary health care settings in low-income regions, with good user satisfaction and potential to improve adherence to evidence-based practices.
Hypertension and diabetes are leading modifiable risk factors for cardiovascular disease worldwide, major contributors to premature disability, and associated with substantial premature death and morbidity . Despite all the advances in the therapy for these diseases, and the fact that many effective treatments are available, there is a great deal of room for improvement. There is a large gap between detection and control of both diseases, and the majority of patients do not reach therapeutic goals [ - ]. The situation is worse in low- and middle-income countries and even worse in rural communities, where awareness, treatment, and control of hypertension are lower than in urban areas [ , , ].
To face this challenge, innovative strategies are required. Clinical decision support systems (CDSSs), which are capable of generating suggestions or information that is specific to individual patients, based on the unique individualized patient information, can make a significant contribution to the effective dissemination of evidence-based practice. They may increase practitioners’ adherence to guidelines, involving clinicians in the translation of research into practice . Therefore, they may increase the proportion of patients who reach blood pressure and glucose goals, with a potential impact in reducing cardiovascular risk.
However, the evidence of impact of CDSSs on key diabetes care outcomes, such as the control of glucose, blood pressure, tobacco use, or appropriate aspirin use has been marginal or inconsistent. One important barrier in this context is CDSS usability . Many CDSSs were not used regularly or on a sustained basis by primary care physicians [ ]. This is true not only for diabetes, but also for CDSS in general. It has been reported that their adoption has been somewhat limited outside of a relatively small number of academic medical centers and integrated health care delivery networks [ , ]. Therefore, in order to translate evidence into practice and develop CDSSs, which can be useful and used in clinical practice, studies that assess usability prior to a large-scale implementation are of utmost importance [ ].
Thus, our aim was to develop a CDSS for diabetes and hypertension management in primary care, to implement it in a low-income and mostly rural region, and to evaluate its usability and health care practitioner satisfaction.
This is a substudy of the HealthRise Brazil Project, a quasi-experimental study, which is part of a multinational study designed to implement and evaluate pilot programs aimed at improving screening, diagnosis, management, and control of hypertension and diabetes among underserved communities . Interventions were designed and implemented in 9 communities in Brazil, India, South Africa, and the United States of America. Each program was designed and implemented by a local grantee and included interventions that were tailored to local needs and contexts [ ]. HealthRise Brazil Project was conducted in 2 centers, Vitória da Conquista, in the state of Bahia, and in the region of Teófilo Otoni (Vale do Mucuri), the state of Minas Gerais. In this region, the project was implemented in 10 municipalities: Frei Gaspar, Ouro Verde de Minas, Crisólita, Catuji, Setubinha, Itaipé, Novo Oriente de Minas, Ladainha, Teófilo Otoni, and Ataleia.
This study was conducted by the HealthRise Brazil Teófilo Otoni team in 4 steps, according to the Medical Research Council framework () [ ]: (1) identification of gaps in usual care; (2) identification of the components of the intervention through discussions with experts; (3) CDSS development and validation; and (4) pilot testing. Theory of change thinking was used throughout the study: programme design, monitoring, and evaluation [ , ].
Identification of Gaps in Usual Care
To evaluate which gaps were already known, we assessed: (1) epidemiological studies and systematic reviews on hypertension and diabetes management; (2) secondary data, including census, Sistema de Informação Ambulatorial (Outpatient Information System), Sistema de Informação da Atenção Básica (Primary Care Information System), Cadastro Nacional de Estabelecimentos de Saúde (National Registry of Health Establishments), Programa Nacional de Melhoria de Acesso e da Qualidade da Atenção Básica (National Program for Improving Access and Quality of Primary Care), social dimensions of the research “Inequalities,” and Pesquisa Nacional de Saúde (National Health Survey). We also assessed information from workshops and focus groups performed by the sponsor, with the participation of municipal and state government health managers, physicians, pharmacists, nurses, nutritionists, physical educators, university professors, specialists, representatives of patients and local communities, community health workers (CHWs), who discussed barriers and access opportunities, as well as priority areas (diagnosis, treatment, and disease management). This information was available the moment we applied for the research funding , and it was the starting point to generate ideas for the intervention design [ ].
Identifying the Components of the Intervention Through Discussion With Experts and Stakeholders
In order to identify the components of the intervention, information was derived from the previous step, as well as previous assessments from our group [, ]; discussions with primary care physicians, cardiologists, endocrinologists, nurses, nutritionists, pharmacists, and physical educators. Besides, meetings and internal workshops were conducted to discuss the topic with primary care practitioners, primary care unit coordinators, local health authorities, and stakeholders. The key stakeholders were identified and involved in the project from the beginning of design of the intervention [ ].
A primary care physician, an internal medicine specialist, a cardiologist, and an endocrinologist discussed the gaps with experts in information technology and stakeholders to identify solutions and components of the intervention, to map anticipated change processes, long-term changes that needed to happen in the target group’s lives, the barriers to those changes, and to explore assumptions and hypotheses .
At that time, all primary care units used a paper-based system to manage patient records; there were different levels of internet connectivity in the primary care centers and health care workers had low technical literacy. All of these factors were taken into account.
Clinical Decision Support System Development and Validation
A team of clinicians established the functional software requirements necessary to record, track, and support decision making for patients with hypertension, diabetes, or both, according to the information obtained from the previous step and also from national and international guidelines [- ] to define the technical specifications for digital systems. A list of indicators was defined for data monitoring.
Development of Content and Functionalities
The technologies used for software development were chosen according to the available technological infrastructure in each municipality. In this context, the following aspects were taken into account: (1) existence of internet connection; (2) type of internet connection; (3) quality of internet connection; (4) existence of mobile and landline telephony; (5) equipment available in health units. Given the different realities and often precarious technology and information, the challenges faced were huge.
For the execution of the project, there was an agreement between the mayor of each municipality, the coordinator of each health unit, and the project management team to guarantee a minimum internet connection and the existence and maintenance of a local server network that would guarantee the registration and access to data while assisting patients. Despite this agreement, and because certain locations had few resources and limited or unstable internet connections, the server system was developed to work locally without the need for an internet connection. The internet was required only to synchronize local servers to the central database, whenever there was a connection, for scientific and monitoring purposes. Therefore, small drops in connection would not lead to any data loss or force an abandonment of using the system. With regard to problems in connectivity and transmission, in cases of momentary loss of synchronization of local data with the centralized data pool, data were synchronized whenever the connection was re-established. In remote areas, we used a portable internet router for transmission, and in rare instances of total absence of internet, which impaired the synchronization of the monthly data monitoring, data were copied to a portable storage device. With data from all units transmitted to the central database, these were extracted and exported to an Excel spreadsheet (Microsoft) for analysis by the research team ().
The application was designed to be used by health care professionals, and consists of: (1) a log-in screen; (2) a patient search screen; (3) a patient registration screen; (4) structured clinical evaluation, clinical decision support, and patient care plan; (5) a health educational groups screen; (6) patient summary screen; and (7) patient management screen ().
The log-in screen allows individualized access to the system through the credentials (user and password) provided to each professional. Each user is registered by a central administrator and each user account is associated with a professional profile that describes the way the application environment looks and operates for that user: physician, nurse, nurse technician, multidisciplinary primary care support team (Núcleo Ampliado de Saúde da Família e Atenção Básica [NASF-AB], explained further in this article), and CHW.
After login, the professional has access to the screen to search for registered patients. If the patient was not registered before, he/she can access the registration screen, to register a new patient (demographic data, address, and telephone number). After registration, the other functionalities become available. Each patient has a specific identification code in the database.
The main functionalities of the application consist of the structured clinical evaluation, the clinical decision support, and the patient care plan. Data requested were manually entered, but data previously entered were saved for the next consultation, for example, medications used. In this case, the health professional could edit the information as necessary. Although few fields are mandatory, the application alerts the health care professional to the importance of completing the information appropriately and completely.
The structured clinical evaluation is shown partially in. It includes data on symptoms, medical history, physical examination (including foot examination in patients with diabetes), current medications, and complementary examination results (laboratory and other tests).
To develop the clinical decision support functionality, recommendations of Brazilian and international evidence-based guidelines assessing hypertension, diabetes, cardiovascular risk, and chronic renal disease were reviewed [- ]. The most relevant clinical recommendations for the management of patients with hypertension, diabetes, or both were derived and organized into a decision algorithm, which bases the CDSS alerts, reminders, and suggestions. In case of conflicting recommendations, the one with the best level of evidence was chosen. When evidence-based guideline recommendations were not available or if they were considered outdated, evidence-based summaries were assessed [ ]. Previous experiences of our group were important for the development of this functionality. Suggestions received about usability in previous opportunities were decisive to improve and create new messages and resources [ , ].
The clinical decision support tab is shown in. In this tab, health care professionals could also access personalized messages to guide the management of each patient, generated according to the data entered in each consultation.
The following variables were calculated using the data entered: (1) BMI, (2) estimated glomerular filtration rate using the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation , and (3) cardiovascular risk based on the Framingham score [ ]. For the alert messages, each one contained a summarized recommendation, tailored according to each patient’s state of health, with auxiliary text containing detailed information and references that support the recommendation.
In the patient care plan, the health care professional can register nonpharmacological interventions, complementary examinations requested, specialist referral and drug prescription (the last 2 functionalities are available only to doctors), and the date for the next consultation. It is also possible to request teleconsultation, defined as a second opinion system that allows an information exchange between distant and local health care professionals, in order to discuss a clinical case when a specialist is not locally available . For this project, it was possible to forward doubts straight to a family physician and an endocrinologist, who were exclusively available for this project, or to other medical and nonmedical specialties, following the workflow of the Telehealth Network of Minas Gerais, a large public Telehealth service which assists 816 municipalities in the state of Minas Gerais [ ]. All teleconsultations were asynchronous. The health care professional could choose which specialty he/she was referring the teleconsultation to, and the normal response time did not exceed 48 hours.
In the educational health groups () screen, the health care professional can register activities performed in different groups in which that patient belongs to (such as hypertension, diabetes, smoking, nutrition).
Patient data obtained in each group (weight, waist circumference, and blood pressure measurements, as well as capillary blood glucose) can be registered, as well as individual absenteeism. There is also a free-text field for any other information that the health professional believes is necessary.
The patient summary screen () shows an overview of the dates of consultations and complementary examinations for each patient.
To conclude the consultation, the health care professional must save the data entered and has the option to print a structured medical record file with all information entered in that consultation, to attach to a physical patient chart. The medical prescription and the referral form can also be printed. Data are recorded in the application and transmitted to a telehealth care center as soon as a stable internet connection was available.
In the patient management screen, it is possible to generate patient lists, according to specific monitoring indicators.
The prototype has been tested multiple times with test cases to ensure that the system was operating as intended, free of bugs, and that the recommendation results matched the prespecified decision tree. A manual insertion of data by 2 physicians, 2 pharmacists, 1 nurse, and 3 medical students with test cases was performed to verify the recommendation response suitability.
Expert Panel Assessment
After adjustments, the prototype was submitted for expert panel assessment, composed of 2 cardiologists, 1 primary care physician, 1 endocrinologist, 1 nurse, 1 pharmacist, and 1 physiotherapist, known as technical reference, who were independent from the project and its implementers, in order to retain a degree of impartiality . The experts used the application for 2 weeks, and were asked to assess functional suitability, stability, appropriateness of CDSS content, clinical benefit, readability, strengths, inconsistencies, perception of usefulness, whether it was appropriate to the local context, and satisfaction with the application. The prototype was then re-adjusted with the necessary changes, according to feedback from the expert panel.
The field study was conducted from October 17 to October 18 in 5 primary care centers in Teófilo Otoni, the main city in the northeast region of Minas Gerais, with 141,934 inhabitants , and 29 primary centers in 9 small towns (with population less than 20,000 inhabitants), as part of the HealthRise Brazil Project—Vale do Mucuri. The region has a low human development: a mean Human Development Index (HDI) of 0.701 for Teófilo Otoni and 0.540-0.595 for the other municipalities, similar to poor African countries. About one-third of the population lives in rural areas (32.3%) [ ]. The HealthRise Brazil Project (2016-2018) consists of 2 parts: patient screening and management (logic model is shown in ).
The intervention was multifaceted, including extensive training of primary care teams, empowerment of patients through education, improvements in access to examinations, strengthening of health educational groups, and implementation of the computer decision support application.
This study is part of patient management. For this substudy, 34 primary health centers in urban and rural areas were included, with family health teams composed of 1 physician and 1 nurse, 1 nurse assistant, and 4 to 6 full-time CHWs . Family health service (FHS) teams provided comprehensive, universal primary care to defined geographical catchment areas, covering populations of up to 1000 households each (between 3000 and 4000 inhabitants), with no overlap or gap between catchment areas. Each FHS team member had defined roles and responsibilities, and national guidelines helped structure FHS responses to most health problems [ , ]. Pharmacists, physiotherapists, nutritionists, psychologists, physical education specialists, and social workers were part of the multidisciplinary primary care support teams (NASF-AB). Each NASF-AB team assisted 3 or 4 primary care centers [ ]. Despite the existence of a university in Teófilo Otoni, clinicians have limited access to specialist referrals and to continuous learning.
Local health authorities and stakeholders were involved in the implementation of the system. The implementation of the application was stepwise, following a schedule of in-person and practical trainings. The participants had the opportunity to handle the devices that would be used. Different training strategies were developed for each professional category. The participants received printed material for further consultation in case of doubt. Furthermore, an online tutorial about the topic was recorded and was available on a platform where they could access online classes about different topics such as hypertension and diabetes management, cardiovascular risk, and operative group planning and organization. Two technology technicians were available online and by telephone to solve doubts and difficulties that arose while using the application. If necessary, they were able to visit each unit.
Fidelity was monitored periodically through checking whether field implementation altered the functionality and stability of the system, changing the intervention from that which was intended, as recommended by the World Health Organization . Malfunctions and other problems with use of the CDSS were quickly fixed.
Feasibility, Usability, and Satisfaction Assessment
A questionnaire evaluating perceived feasibility, usability, and utility of the application and professionals’ satisfaction, previously developed by our group , was applied after 6 months. The first part of the questionnaire included participant characteristics: age, specialty, time since graduation, time working in primary care, previous knowledge in dealing with health technologies, previous access to the internet, and previous access to continued education. The second part included Likert scale questions, varying from 1 (strongly disagree) to 5 (strongly agree), to assess feasibility, usability, and satisfaction. This instrument was evaluated in its reliability capacity applied in this context.
At the end of the HealthRise Project, 2 focus groups with primary care physicians and nurses were performed. The health care professionals talked about the advantages of using the software and what they thought should be improved. The sessions lasted for 1.5 hours and were audio-recorded. Questions started out broadly and became more specific as participants felt more comfortable and opened up.
Distributions of continuous variables were examined for normality by using Kolmogorov–Smirnov tests. As the distributions were asymmetrical, these variables were expressed as median and corresponding IQR. Categorical variables were presented as frequencies and proportions. The reliability of the questionnaire to evaluate the perceived feasibility, usability, and utility was evaluated using Cronbach α. Data analyses were performed using the software IBM SPSS Statistics for Windows, version 21.0 (IBM Corp.).
Ethical approval was obtained from the Universidade Federal dos Vales do Jequitinhonha e Mucuri Research Ethics Committee (number 65808517.9.0000.5108). Written and informed consent was obtained from all participants.
Phase I: CDSS Development and Validation
An initial bank of 168 reminders and suggestions was created for the clinical decision support functionality. After message content refinement, a final bank of 159 reminders and suggestions was used.shows examples of the CDSS messages. As there were many messages, we choose to extensively test 26 messages first; we implemented the application with those messages, and continued testing the other messages extensively. We divided the messages in blocks of 10, ranked the groups according to the priority of the recommendations for diabetes and hypertension management, and tested each group before implementation.
Examples of clinical decision support system reminders and suggestions. ACE: angiotensin-converting enzyme.
- “Reinforce the importance of adherence to treatment and participation in the educational group. Request a visit from the community health worker”.
- “Cardiovascular risk greater than 20% in 10 years. The prescription of statin is suggested.”
- “Do not prescribe thiazide diuretic for this patient due to reduced renal function.”
- “If you decide to add another anti-hypertensive, choose one of the first-line groups (ACE inhibitor, angiotensin receptor II blocker, thiazide diuretic, long-acting calcium channel blocker), taking the contraindications into account.”
- “Metformin is being used at a dosage above the maximum recommendation. Dose reduction is suggested.”
- “This patient did not have a fundoscopic assessment in the last year”.
- “It is suggested to schedule the next consultation in one month, to reassess blood pressure levels”.
During internal tests, most errors identified were due to misplaced parenthesis and connectors in the commands of the decision algorithm. Posterior manual insertion of data evidenced remaining errors, spelling mistakes, and inconsistencies that were properly revised before the test by the panel of specialists. The expert panel–suggested changes were used to enhance the application.
Phase II: Feasibility, Usability, and Satisfaction Assessment
At the 6-month assessment, there were 1939 patients registered in the application database and 2160 consultations were performed by primary care teams. Of the 96 health care professionals who were invited for the usability assessment, 26% (25/96) were physicians, 46% (44/96) were nurses, and 28% (27/96) were from other health professionals. The number of participants per town varied from 5 (in Crisólita) to 14 (in Ataleia). Participant characteristics are shown in. Age varied from 23 to 68 and time since graduation varied from 1 to 29 years. Nurses and other health professionals were predominately women.
|Variable||Physicians (n=25)||Nurses (n=44)||Other health professionals (n=27)||Total|
|Age (years), median (IQR)||35 (30-43)||33 (30-36)||32 (27-36)||33 (30-36)|
|Female sex, n (%)||11 (44)||33 (75)||22 (81)||66 (69)|
|Time since graduation (years), median (IQR)||9.0 (7.0-11.0)||9.0 (6.5-11.0)||5.5 (3.0-9.0)||8.0 (4.3-10.8)|
|Time working in primary care (years), median (IQR)||7.0 (4.0-16.0)||5.5 (2.0-9.4)||3.0 (1.0-4.0)||5.0 (2.0-9.0)|
|Self-reported knowledge in information technology, n (%)|
|Excellent||9 (36)||10 (23)||7 (26)||26 (27)|
|Good||9 (36)||24 (55)||15 (56)||48 (50)|
|Satisfactory||7 (28)||9 (20)||5 (19)||21 (22)|
|Inadequate||0 (0)||1 (2)||0 (0)||1 (1)|
|Use of any form of technology for work before this research project (yes), n (%)||12 (48)||31 (70)||19 (70)||62 (65)|
|Computer available in the workplace for routine use before the research project (yes), n (%)||11 (44)||32 (73)||19 (70)||62 (65)|
|Internet access in the workplace (yes), n (%)||20 (80)||39 (89)||23 (85)||82 (85)|
|Internet use frequency (yes), n (%)|
|Daily||21 (84)||40 (91)||22 (81)||83 (86)|
|Almost daily||2 (8)||3 (7)||4 (15)||9 (9)|
|Weekly||2 (8)||0 (0)||0 (0)||2 (2)|
|Rarely||0 (0)||1 (2)||0 (0)||1 (1)|
|Participated in an updating activity on management of hypertension, diabetes mellitus, or cardiovascular risk in the last year (yes), n (%)||17 (68)||18 (41)||7 (26)||42 (44)|
The results of the questionnaire including 24 items on impressions of feasibility, usability, and utility are presented in. The reliability analysis of the instrument and 4 dimensions, applied to this population, presented a global Cronbach α of .93 ( and detailed in ), showing adequate internal consistency of this instrument for evaluation of the perceived feasibility, usability, and utility of the CDSS.
|Item||Physicians (n=25)||Nurses (n=44)||NASF-ABc (n=27)||Total|
|Feasibility, median (IQR)|
|The application can be used in primary care setting||4 (4-5)||5.00 (4-5)||4 (4-5)||4 (4-5)|
|It is easy to be incorporated in work routine||4 (2-5)||4 (4-4)||4 (3-4)||4 (3-4)|
|It does not cause significant delays in daily routine||2 (2-3)||3 (2-4)||4 (3-4)||3 (2-4)|
|Usability, median (IQR)|
|Overall, evaluation is good||4 (3.25-4.75)||4 (4-5)||4 (4-5)||4 (4-5)|
|The screens are easy to understand||4 (4-4)||4 (4-5)||4 (4-4.5)||4 (4-5)|
|I was able to find the information I was looking for||4 (4-4)||4 (4-4)||4 (4-5)||4 (4-4)|
|The definitions of comorbidities are clear||4 (4-4)||4 (4-4.75)||4 (4-5)||4 (4-4)|
|The fields are easy to fill||4 (4-4)||4 (4-5)||4 (4-5)||4 (4-5)|
|It has proper interface||4 (4-4)||4 (4-5)||4 (4-5)||4 (4-5)|
|It is stable during the use||4 (2-4)||3 (2-4)||3 (2-4)||3 (2-4)|
|Utility, median (IQR)|
|The application might improve patient care||4 (4-5)||5 (4-5)||5 (4-5)||4 (4-5)|
|The recommendations brought new information on hypertension, diabetes mellitus, and cardiovascular risk||4 (4-4)||4 (4-5)||4 (4-5)||4 (4-5)|
|It was useful to calculate the cardiovascular risk||4 (4-4.75)||4 (4-5)||4 (4-5)||4 (4-5)|
|It was useful to promote cardiovascular disease prevention||4 (4-5)||4 (4-5)||4 (4-5)||4 (4-5)|
|It assisted me to treat my patients||4 (4-4)||4 (4-5)||4 (4-4)||4 (4-5)|
|It assisted me to choose complementary examinations||4 (3-4)||4 (4-5)||4 (3.75-4.25)||4 (4-5)|
|It was helpful to decrease referral to specialists||4 (2-4)||4 (3-4)||4 (4-4)||4 (3-4)|
|It was useful for different professional categories||4 (3-4)||4 (4-4)||4 (4-4)||4 (4-4)|
|I believe the recommendations are appropriate||4 (3-4)||4 (4-4)||4 (3.75-4.25)||4 (4-4)|
|I used the suggestions to modify patient care||4 (4-4)||4 (4-5)||4 (4-5)||4 (4-5)|
|Satisfaction, median (IQR)|
|Overall, I am satisfied with the application||4 (4-4.75)||4 (4-5)||4 (4-5)||4 (3-4)|
|It may be beneficial for patient care||4 (4-4.75)||4 (4-5)||4 (4-5)||4 (4-5)|
|I will keep using it for patient care||4 (4-5)||4 (4-5)||4 (4-5)||4 (4-5)|
|I would recommend this application||4 (3-4.75)||4 (4-5)||4 (4-5)||4 (4-5)|
aRanging from 1 to 5.
bCronbach α for the Global questionnaire (24 items) is .93.
cNASF-AB: Núcleo Ampliado de Saúde da Família e Atenção Básica (multidisciplinary primary care support teams).
At the end-line assessment, health care professionals registered 4211 patients in the application’s database and 7960 consultations were performed by primary health care teams. Patients were predominantly female (2819/4211, 66.94%), median age was 55.0 (IQR 47.0-62.0). A total of 3993 (94.82%) patients were diagnosed with hypertension and 1028 (24.41%) were diagnosed with diabetes and there were 810 (19.24%) patients with diagnosis of both hypertension and diabetes.
In the focus groups, 13 physicians and 4 nurses participated (71% [12/17] female, median age 34 [IQR 31-39], ranging from 24 to 69 years). With regard to the difficulties dealing with the technology, the majority of participants reported some difficulties in the first days using the software, but no problems afterward. However, there was a statement of more difficulties among older professionals, who received help from the younger team members:
I started handling the software for patient assistance, as a nurse. I had no difficulties. Nowadays I help other professionals, such as doctors, who are more used to pen and paper. My unit has a doctor with 30 years experience, who had more difficulty adapting (using the software). As I showed him step by step and all functionalities of the software, he became interested and found it cool When he learnt that he could write patient prescriptions and print it using the software, he was very satisfied. So now, he prints the patient prescription and the exams, correctly. This way, he saves a lot of time.
[Female, 37-year-old nurse, 13 years’ experience working as a primary care nurse]
The structured clinical evaluation was perceived as an advantage, as exemplified below:
The questions we always ask (using the software). Are you feeling shortness of breath? Do you wake up out of breath? Sometimes in a rush, we forget (to ask those questions to the patient). But we always ask them (when using the software). It is as if you take the query and say: this is indispensable for you to ask to the patient. You always have to remember to ask about it. So, for me it's great. I will always remember to give counselling, for example, when the patient is a smoker, at the end there is always the ‘Did you talk about smoking?’ reminder. So there you will remember to tell the patient “Look, you have to stop smoking”. For me, this is very cool.
[Female, 31-year-old physician, 4.5 years’ experience working as a primary care physician]
There was no habit of measuring the blood pressure sitting, lying down and standing. After we started using the software, it became routine. We do not need to say anything. The nurse technician already measures the blood pressure of all patients that way.
[Female, 31-year-old physician, 4.5 years’ experience working as a primary care physician]
However, the time spent in the consultation was seen as a challenge for some participants. In some cases, they were forced by the health department to attend each patient for a maximum of 15 minutes, but this time was not sufficient when attending the patients using the software:
I take an average of 30 minutes per consultation, i.e. two consultations. This is the same I do with prenatal consultations for pregnant women.
[Male, 39-year-old physician, 6 years’ experience working as a primary care physician]
In other cases, there was an organizational issue in the primary care, which made it difficult to attend patients using the software:
My primary care center works as an emergency care. So we set aside a day to attend the patients from the project, which is a longer consultation. However, for example, we book ten patients from the program, but 20 show up; six from the project and the rest are patients with acute diseases. It is a challenge to attend the patients using the software, as there are other patients waiting. That makes it difficult.
[Female, 32-year-old physician, 6 years’ experience working as a primary care physician]
Often the CHWs were the ones who registered the patients in the software and their educational level was usually low in that region, so some health care professionals interviewed complained that spelling mistakes in patient names made them waste time trying to guess how the CHWs might have registered the patient’s name.
With regard to the clinical decision support functionality, all participants were consistent to affirm they were very satisfied with it. They considered the alerts and recommendations very useful for patient care, and reported that it influenced their patient care plan.
We usually don’t have time to calculate cardiovascular risk and kidney function. The software helps a lot.
[Male, 39-year-old physician, 6 years’ experience working as a primary care physician]
When attending a patient with high cardiovascular risk, who was taking 40 milligrams of simvastatin, and also amlodipine for hypertension, I received an alert to adjust the dosage of simvastatin to 20 milligrams, as amlodipine may increase the serum concentration of simvastatin and thus increase the risk of adverse effects. I had no idea, I learnt with the alert and changed patient prescription.
[Female, 31-year-old physician, 4.5 years’ experience working as a primary care physician]
There was a suggestion to improve the clinical decision support functionality, creating “red flags”:
I think the abnormal findings or the alerts could be in red. For example, the alert ‘This patient is in stage 3a renal failure’ should be in red.
[Female, 39-year-old physician, 6 years’ experience working as a primary care physician]
Concerning the ability to perform teleconsultations, the majority of participants said they have no time for it during the working hours. The ones who do it usually have to use their off-working hours:
I love to do teleconsultations, it helps the patients a lot. So much so that I am currently second (professional who requested the highest number of teleconsultations) in the state of Minas. We can help a lot people in primary care (using the teleconsultations). But I do it (the teleconsultation) at home; That's why it's hard work and a lot of people don't do it, there's no way to do it on the job.
[Male, 69-year-old physician, 29 years’ experience working as a primary care physician]
Finally, each software screen was projected and we discussed which variables they felt could be removed. All participants were consistent to affirm that all variables were very useful, except for patient admission, in which they had to fill in the date of admission and patient discharge, as well as the reason for admission. They considered these variables were not necessary, and their work could be reduced if they could be removed.
This study described the development, usability, and satisfaction assessment of a web-based CDSS for diabetes, hypertension, and cardiovascular risk management in primary care. Its implementation in a low-resource setting, for users who were used to the paper-based patient records only, has shown to be feasible. Participants with a wide age range and experience in primary care found it to be usable, reported satisfaction, and attended almost 2000 patients using the system over 6 months and over 4000 patients by the end of follow-up.
With the adoption of the 2030 Agenda for Sustainable Development and the 17 Sustainable Development Goals , health systems and stakeholders worldwide are interested in innovative approaches to achieving universal health coverage objectives [ , ]. Although the Brazilian Constitution states that health care is free and provided by the public health system to all citizens, patients have unequal access to specialized health care services, especially in remote municipalities. Primary care physicians are often young people (less than 35 years of age) with little professional experience (less than 10 years of medical practice) [ ]. Additionally, in those municipalities, there is a strong sense of professional isolation and high turnover of health professionals, which compromises the quality of health care [ ]. In a recent meta-analysis, blood pressure control rates ranged from 43.7% to 67.5% in Brazilian patients [ ]. Brazil is also one of the “most important examples of the alarming picture of diabetes in emergent societies,” with the fourth largest number of people with diabetes, and a poorer glycemic control rate than that observed in Europe and the United States. Even when a less stringent target was considered, only 48.5% of patients had an HbA1c level less than 8% [ ]. In this context, digital health interventions may be very useful, especially those focusing on existing evidence-based health interventions, on the determinant layers of universal health coverage [ ].
As CDSS recommendations evolve over time as evidence and the patient’s clinical state evolves, it provides a dynamic and personalized patient care plan that can be easily accessed by any health team member at any point in time , and may reduce the likelihood of unhelpful or risky prescriptions [ ]. In this study, CDSS was implemented as part of a multifaceted strategy to enhance the delivery of evidence-based care in a resource-constrained area. It is innovative, as the majority of previous studies assessed the use of CDSS for hypertension and diabetes management in high-income countries [ ]. Therefore, there is paucity of data on low- and middle-income countries and rural communities, which are the areas in most need and where the access to information technologies may be impaired. In our study, one-third of the total sample (34/96, 35%) and 52% of the doctors (13/25) did not use any form of technology for work before this research project. They did not use any kind of computerized physician order entry or electronic health records, so the implementation of the CDSS brought a significant change in workflow, which could be an important barrier for its usability [ , ]. Therefore, motivating these health care professionals and providing training in using the CDSS were challenging. Developing user-friendly interfaces and avoiding unnecessary information and excess of clicks were essential. Participants agreed that the interface was suitable. Making a tutorial video was important, but it did not replace the need for in-person training, and technical support was of utmost importance. Although the expert panel found the application intuitive, with no need for previous training, the health care professionals did not agree with that, which suggests different levels of technological skills.
With regard to the change in the workflow, health care professionals had to enter all the data items manually. It is reported that the burden of data entry may make them give up and abandon the CDSS . At the same time, there is evidence that arduous data entry facilities adversely affect clinicians’ satisfaction [ ], but the specificity level of computer-generated advice is known to highly influence the chance that physicians adhere to the advice [ ]. To provide specific advice, is not possible to rely on limited data. A solution we found to try to overcome those barriers, facilitating the integration in the workflow and avoiding duplication of work, was to enable the health care professional to print the consultation using the CDSS and attach it to patient charts. The fact that it could be used by all primary care practitioners, each category with its allowances, and one complementing the other, with the inclusion of the educational groups and a patient management screen, was also important. Additionally, all the data on comorbidities and medications from a previous consultation did not need to be filled in again. Although the majority of primary care practitioners reported satisfaction with the application, doctors gave a low score for the variable “significant delays in daily routine” (median 2.0, IQR 2.0-3.0). Although the median score for doctors for “easy to be incorporated in work routine” was 4.0, 30% (7/23) of them choose a lower score (P25 [first quartile]=2.0). This was also reported in the focus groups. By contrast, when asked about reducing variables in the application, all professionals were consistent with the request to keep it the way it was.
Although the clinical impact assessment was not the focus at the early stages of implementation, there was improvement in outcomes related to hypertension and diabetes management, which are shown in another publication . It is well established that managing the global cardiovascular risk is more important than only reducing blood pressure, glucose levels, or both [ ], and this cardiovascular risk should drive preventive strategies. However, the majority of published CDSS did not include the assessment of cardiovascular risk. Choosing the cardiovascular risk calculator to include in the software was challenging, as no score was validated for the South American population [ ]. We decided to use the Framingham score [ ], as it was widely validated in different populations (European, Asian, Australian, and North American) and assessed important outcomes (coronary artery disease, stroke, peripheral artery disease, and heart failure) with no limitation in number of events (as in the case with the Atherosclerotic Cardiovascular Disease Risk Algorithm [ASCVD], which assessed only the first event) or restricted to fatal events only (as in the case with the Systematic Coronary Risk Evaluation [SCORE]) [ ]. Additionally, it includes a number of variables that are easily obtained. In case total cholesterol and high-density lipoprotein cholesterol levels are not available yet, the risk can be estimated using another version of the score, in which BMI replaces the lipid levels [ , ].
This is study has some limitations. As was determined by the Ethics Committee, responses to the questionnaire were not identified, so we were not able to assess the results of feasibility, usability, and satisfaction according to the number of patients managed by each user. Additionally, there was no formal evaluation of the effectiveness. The software was part of a multifaceted intervention, whose results were published elsewhere , and it is not possible to access the impact of the software by itself. Meta-analyses suggest that CDSS with recommendations to both patients and health care professionals is more effective than providing CDSS to health care professionals only [ , ]. In this study, due to the high levels of illiteracy, we opted not to work directly with patients at that time.
The majority of health care professionals (52/96, 54%) and 32% (8/25) of the doctors had no access to update an activity in the previous year. It is well established that CDSSs are designed to support decision making rather than making the decisions for the user, who makes the final decision based on his/her knowledge, experience, and scientific evidence provided by the CDSS or other sources . Therefore, another important step was clinical training. Professionals had access to online classes, which were designed for each professional category and took into account their reality in terms of resources. Therefore, we believe that the opportunity of updating knowledge via online classes and by using the CDSS may have been one success factor, as has been already reported in the literature [ , ]. For those health care workers who had impaired access to continued education, there was a perception that the CDSS was a useful tool to improve patient care. It may have overcome the delays in daily routine and the perceived difficulty to be incorporated into the clinical workflow.
Management of diabetes and hypertension is a team effort, and many health care professionals may be involved [, ], so we believe that the fact that all health care practitioners could access the CDSS, each on in their own role, was another important success factor, avoiding keeping patient care completely medical centered.
Careful monitoring of CDSS use rates, requesting feedback from users, and adjusting the application accordingly were other important success factors. Interventions lacking robust monitoring activities are unlikely to generate the impact expected from them and may lead to misjudgment of the digital health intervention as being not effective. As recommended by the WHO, robust monitoring and evaluation plans are essential to support effective implementation and potential intervention scale-up .
It is known that the level of adoption of a digital health intervention by users is “dependent on the end-users’ interaction with the technology and their belief/opinion that use of the technology will benefit their health” . The responses to the questionnaire and the comments in the focus groups showed that this was probably another success factor in our study.
To ensure sustainability, the Brazilian Ministry of Health, MG state government, and municipal governments were involved from the beginning, as well as different stakeholders and local health professionals. The successful experience caught the Brazilian Ministry of Health’s interest, who in turn sponsored the continuation of the project’s activities for 2 years in the same region.
We are planning to expand the project to primary care units in other towns, to include other diseases, and to implement in other settings. Our team is now working on the intervention refinement and software improvement, to subsequently conduct a cluster randomized controlled study to test the intervention implementation in other settings. The integration of the CDSS with the incipient electronic medical record of the Brazilian public health system (e-SUS) is a challenge, but we are currently working to overcome this barrier. Although there is limited transferability of the results to other settings, the lessons learnt may be useful for the implementation of other CDSSs in Brazil and around the world.
In this study, a CDSS developed to assist the management of patients with hypertension or diabetes or both was applicable in the context of primary health care setting in a low income region, with good user’s satisfaction and potential to improve adherence to evidence-based practices.
HealthRise Brazil Project was designed in partnership with Medtronic Foundation (sponsoring institution), Abt Associates, and Institute for Health Metrics and Evaluation. Those institutions had no influence on software development, implementation, and assessment. AR was supported in part by the Brazilian research agencies CNPQ (bolsa de produtividade em pesquisa 310679/2016-8 and IATS 465518/2014-1) and FAPEMIG (Programa Pesquisador Mineiro PPM-00428-17 and Rede de Teleassistência de Minas Gerais RED-00081-16). AR and MM are members of the National Institute of Science and Technology for Health Technology Assessment (IATS/CNPq). JO received doctoral grant (number 88887.142558/2017-00) from the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) – Finance Code 001, as part of the IATS/CNPq. KA received grants from the IATS (number 372857/2019-1). The authors thank Santino Filoso for revising the manuscript.
AR, CC, VP, and MM were responsible for the research protocol and coordinated the study. MM, JO, JM, LB, and TS participated in the CDSS development and testing. CC, JO, JM, LC, LR, MM, TS and VP participated in the CDSS implementation. CC was responsible for coordination of the research team for the application of the questionnaire. MM performed the data analysis and drafted the manuscript. All other authors reviewed and edited the manuscript. JO organized the manuscript references. All authors approved the final version.
Conflicts of Interest
Supplemental figure showing the dynamics of the system.PNG File , 95 KB
Supplemental figure showing the logic model of the study.PNG File , 1775 KB
Supplemental table with detailed data of feasibility, usability and satisfaction assessment scores.DOCX File , 22 KB
- Williams B. High blood pressure in young people and premature death. BMJ 2011 Feb 22;342(feb22 2):d1104-d1104. [CrossRef] [Medline]
- Chow CK, Teo KK, Rangarajan S, Islam S, Gupta R, Avezum A, PURE (Prospective Urban Rural Epidemiology) Study investigators. Prevalence, awareness, treatment, and control of hypertension in rural and urban communities in high-, middle-, and low-income countries. JAMA 2013 Sep 04;310(9):959-968. [CrossRef] [Medline]
- Benjamin E, Blaha M, Chiuve S, Cushman M, Das SR, Deo R, American Heart Association Statistics CommitteeStroke Statistics Subcommittee. Heart Disease and Stroke Statistics-2017 Update: A Report From the American Heart Association. Circulation 2017 Mar 07;135(10):e146-e603 [FREE Full text] [CrossRef] [Medline]
- Blonde L, Aschner P, Bailey C, Ji L, Leiter LA, Matthaei S, Global Partnership for Effective Diabetes Management. Gaps and barriers in the control of blood glucose in people with type 2 diabetes. Diab Vasc Dis Res 2017 May;14(3):172-183 [FREE Full text] [CrossRef] [Medline]
- Khunti K, Gomes M, Pocock S, Shestakova MV, Pintat S, Fenici P, et al. Therapeutic inertia in the treatment of hyperglycaemia in patients with type 2 diabetes: A systematic review. Diabetes Obes Metab 2018 Feb;20(2):427-437 [FREE Full text] [CrossRef] [Medline]
- Fox J, Patkar V, Chronakis I, Begent R. From practice guidelines to clinical decision support: closing the loop. J R Soc Med 2009 Nov;102(11):464-473 [FREE Full text] [CrossRef] [Medline]
- O'Connor P, Sperl-Hillen J, Fazio C, Averbeck B, Rank B, Margolis K. Outpatient diabetes clinical decision support: current status and future directions. Diabet Med 2016 Jun;33(6):734-741 [FREE Full text] [CrossRef] [Medline]
- Roshanov P, Fernandes N, Wilczynski J, Hemens BJ, You JJ, Handler SM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ 2013 Feb 14;346:f657 [FREE Full text] [CrossRef] [Medline]
- Jaspers M, Smeulers M, Vermeulen H, Peute L. Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. J Am Med Inform Assoc 2011 May 01;18(3):327-334 [FREE Full text] [CrossRef] [Medline]
- Wright A, Sittig D, Ash J, Sharma S, Pang J, Middleton B. Clinical decision support capabilities of commercially-available clinical information systems. J Am Med Inform Assoc 2009;16(5):637-644 [FREE Full text] [CrossRef] [Medline]
- Kawamoto K, Lobach D. Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior. AMIA Annu Symp Proc 2003:361-365 [FREE Full text] [Medline]
- Medtronic Foundation, Abt Associates, The Institute for Health Metrics and Evaluation. HealthRise Final Report: Expanding Access to Chronic Disease Care through Community Approaches in Four Countries.: Medtronic Foundation; 2019. URL: http://www.foundation.medtronic.com/content/dam/medtronic_foundation/our_impact/HealthRise%20Final%20Report_FINAL.pdf [accessed 2019-11-22]
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008 Sep 29;337:a1655 [FREE Full text] [CrossRef] [Medline]
- Vogel I. Review of the use of 'Theory of Change' in international development. London: DFID; 2012. URL: http://assets.publishing.service.gov.uk/media/57a08a5ded915d3cfd00071a/DFID_ToC_Review_VogelV7.pdf [accessed 2019-11-23]
- World Health Organization. Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. Geneva; 2016. URL: http://apps.who.int/iris/bitstream/handle/10665/252183/9789241511766-eng.pdf [accessed 2019-11-23]
- World Health Organization. A guide to implementation research in the prevention and control of noncommunicable diseases. Geneva; 2016. URL: http://apps.who.int/iris/bitstream/10665/252626/1/9789241511803-eng.pdf [accessed 2019-11-26]
- Maia JX, de Sousa LAP, Marcolino MS, Cardoso CS, da Silva JLP, Alkmim MBM, et al. The Impact of a Clinical Decision Support System in Diabetes Primary Care Patients in a Developing Country. Diabetes Technol Ther 2016 Apr;18(4):258-263. [CrossRef] [Medline]
- Silveira D, Marcolino M, Machado E, Ferreira CG, Alkmim MBM, Resende ES, et al. Development and Evaluation of a Mobile Decision Support System for Hypertension Management in the Primary Care Setting in Brazil: Mixed-Methods Field Study on Usability, Feasibility, and Utility. JMIR Mhealth Uhealth 2019 Mar 25;7(3):e9869 [FREE Full text] [CrossRef] [Medline]
- American Diabetes Association. Standards of Medical Care in Diabetes - 2017. Diabetes Care 2017;40(Suppl. 1):S1-S135.
- Whelton P, Carey R, Aronow W, Casey DE, Collins KJ, Dennison Himmelfarb C, et al. 2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA Guideline for the Prevention, Detection, Evaluation, and Management of High Blood Pressure in Adults: Executive Summary: A Report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. Hypertension 2018 Jun;71(6):1269-1324. [CrossRef] [Medline]
- Mancia G, Fagard R, Narkiewicz K, Redón J, Zanchetti A, Böhm M, Task Force Members. 2013 ESH/ESC Guidelines for the management of arterial hypertension: the Task Force for the management of arterial hypertension of the European Society of Hypertension (ESH) and of the European Society of Cardiology (ESC). J Hypertens 2013 Jul;31(7):1281-1357. [CrossRef] [Medline]
- James P, Oparil S, Carter B, Cushman WC, Dennison-Himmelfarb C, Handler J, et al. 2014 evidence-based guideline for the management of high blood pressure in adults: report from the panel members appointed to the Eighth Joint National Committee (JNC 8). JAMA 2014 Feb 05;311(5):507-520. [CrossRef] [Medline]
- KDIGO. Clinical Practice Guideline for the Evaluation and Management of Chronic Kidney Disease. Kidney International Supplements 2013;3:1-150 [FREE Full text]
- Leung AA, Nerenberg K, Daskalopoulou SS, McBrien K, Zarnke KB, Dasgupta K, CHEP Guidelines Task Force. Hypertension Canada's 2016 Canadian Hypertension Education Program Guidelines for Blood Pressure Measurement, Diagnosis, Assessment of Risk, Prevention, and Treatment of Hypertension. Can J Cardiol 2016 May;32(5):569-588. [CrossRef] [Medline]
- Malachias MVB, Souza WKSB, Plavnik FL, et al. 7th Brazilian Guideline of Arterial Hypertension. Arq Bras Cardiol 2016;107(3 (Suppl.3)):1-83 [FREE Full text]
- Meireles A, Alves JA, Galil A, et al. Linha-guia de Hipertensão Arterial Sistêmica, Diabetes Mellitus e Doença Renal Crônica. Belo Horizonte: Secetaria de Estado de Saúde de Minas Gerais; 2013. URL: http://www.saude.mg.gov.br/images/documentos/guia_de_hipertensao.pdf [accessed 2018-04-10]
- Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Estratégias para o cuidado da pessoa com doença crônica: diabetes mellitus. Brasília; 2013. URL: http://126.96.36.199/dab/docs/portaldab/publicacoes/caderno_36.pdf [accessed 2018-04-10]
- Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Estratégias para o cuidado da pessoa com doença crônica: hipertensão arterial. Brasília; 2013. URL: http://188.8.131.52/dab/docs/portaldab/publicacoes/caderno_37.pdf [accessed 2018-04-10]
- Reiner Z, Catapano AL, De Backer G, Graham I, Taskinen MR, Wiklund O, ESC Committee for Practice Guidelines (CPG) 2008-20102010-2012 Committees. ESC/EAS Guidelines for the management of dyslipidaemias: the Task Force for the management of dyslipidaemias of the European Society of Cardiology (ESC) and the European Atherosclerosis Society (EAS). Eur Heart J 2011 Jul;32(14):1769-1818. [CrossRef] [Medline]
- Dicenso A, Bayley L, Haynes RB. Accessing pre-appraised evidence: fine-tuning the 5S model into a 6S model. Evid Based Nurs 2009 Oct;12(4):99-101. [CrossRef] [Medline]
- Levey A, Stevens L, Schmid C, Zhang YL, Castro AF, Feldman HI, CKD-EPI (Chronic Kidney Disease Epidemiology Collaboration). A new equation to estimate glomerular filtration rate. Ann Intern Med 2009 May 05;150(9):604-612 [FREE Full text] [CrossRef] [Medline]
- D'Agostino RS, Vasan R, Pencina M, Wolf PA, Cobain M, Massaro JM, et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation 2008 Feb 12;117(6):743-753. [CrossRef] [Medline]
- Nerlich M, Balas E, Schall T, Stieglitz SP, Filzmaier R, Asbach P, G8 Global Health Applications Subproject 4. Teleconsultation practice guidelines: report from G8 Global Health Applications Subproject 4. Telemed J E Health 2002;8(4):411-418. [CrossRef] [Medline]
- Soriano Marcolino M, Minelli Figueira R, Pereira Afonso Dos Santos J, Silva Cardoso C, Luiz Ribeiro A, Alkmim MB. The Experience of a Sustainable Large Scale Brazilian Telehealth Network. Telemed J E Health 2016 Nov;22(11):899-908. [CrossRef] [Medline]
- Instituto Brasileiro de Geografia e Estatística. Estatísticas sociais por cidade e estado. Teófilo Otoni, Minas Gerais.: IBGE; 2018. URL: http://cidades.ibge.gov.br/brasil/mg/teofilo-otoni/panorama [accessed 2018-04-10]
- Instituto Brasileiro de Geografia e Estatística. População residente, por situação do domicílio, sexo e idade, segundo a condição no domicílio e compartilhamento da responsabilidade pelo domicílio.: IBGE; 2019. URL: http://sidra.ibge.gov.br/Tabela/1378#resultado [accessed 2019-09-18]
- Macinko J, Harris M. Brazil's family health strategy--delivering community-based primary care in a universal health system. N Engl J Med 2015 Jun 04;372(23):2177-2181. [CrossRef] [Medline]
- Macinko J, Harris M, Rocha M. Brazil's National Program for Improving Primary Care Access and Quality (PMAQ): Fulfilling the Potential of the World's Largest Payment for Performance System in Primary Care. J Ambul Care Manage 2017;40 Suppl 2 Supplement, The Brazilian National Program for Improving Primary Care Access and Quality (PMAQ):S4-S11 [FREE Full text] [CrossRef] [Medline]
- United Nations. Transforming our world: the 2030 Agenda for Sustainable Development. New York: United Nations; 2015. URL: http://sustainabledevelopment.un.org/content/documents/21252030%20Agenda%20for%20Sustainable%20Development%20web.pdf [accessed 2019-11-25]
- Kieny M, Bekedam H, Dovlo D, Fitzgerald J, Habicht J, Harrison G, et al. Strengthening health systems for universal health coverage and sustainable development. Bull World Health Organ 2017 Jul 01;95(7):537-539 [FREE Full text] [CrossRef] [Medline]
- Miotto B, Guilloux A, Cassenote A, Mainardi G, Russo G, Scheffer M. Physician's sociodemographic profile and distribution across public and private health care: an insight into physicians' dual practice in Brazil. BMC Health Serv Res 2018 Apr 23;18(1):299 [FREE Full text] [CrossRef] [Medline]
- Paim J, Travassos C, Almeida C, Bahia L, Macinko J. The Brazilian health system: history, advances, and challenges. Lancet 2011 May 21;377(9779):1778-1797. [CrossRef] [Medline]
- Picon R, Dias-da-Costa J, Fuchs F, Olinto M, Choudhry N, Fuchs S. Hypertension Management in Brazil: Usual Practice in Primary Care-A Meta-Analysis. Int J Hypertens 2017;2017:1274168 [FREE Full text] [CrossRef] [Medline]
- de Almeida-Pititto B, Dias M, de Moraes ACF, Ferreira S, Franco D, Eliaschewitz F. Type 2 diabetes in Brazil: epidemiology and management. Diabetes Metab Syndr Obes 2015;8:17-28 [FREE Full text] [CrossRef] [Medline]
- Koopman RJ, Kochendorfer KM, Moore JL, Mehr DR, Wakefield DS, Yadamsuren B, et al. A diabetes dashboard and physician efficiency and accuracy in accessing data needed for high-quality diabetes care. Ann Fam Med 2011;9(5):398-405 [FREE Full text] [CrossRef] [Medline]
- Jia P, Zhao P, Chen J, Zhang M. Evaluation of clinical decision support systems for diabetes care: An overview of current evidence. J Eval Clin Pract 2019 Feb;25(1):66-77. [CrossRef] [Medline]
- Castillo R, Kelemen A. Considerations for a successful clinical decision support system. Comput Inform Nurs 2013 Jul;31(7):319-26; quiz 327. [CrossRef] [Medline]
- Shiffman R, Liaw Y, Brandt C, Corb G. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 1999;6(2):104-114 [FREE Full text] [CrossRef] [Medline]
- Stewart J, Manmathan G, Wilkinson P. Primary prevention of cardiovascular disease: A review of contemporary guidance and literature. JRSM Cardiovasc Dis 2017;6:2048004016687211 [FREE Full text] [CrossRef] [Medline]
- Damen J, Pajouheshnia R, Heus P, Moons KGM, Reitsma JB, Scholten RJPM, et al. Performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis. BMC Med 2019 Jun 13;17(1):109 [FREE Full text] [CrossRef] [Medline]
- Gaziano T, Young C, Fitzmaurice G, Atwood S, Gaziano J. Laboratory-based versus non-laboratory-based method for assessment of cardiovascular disease risk: the NHANES I Follow-up Study cohort. Lancet 2008 Mar 15;371(9616):923-931 [FREE Full text] [CrossRef] [Medline]
- Collins S, Drew P, Watt I, Entwistle V. 'Unilateral' and 'bilateral' practitioner approaches in decision-making about treatment. Soc Sci Med 2005 Dec;61(12):2611-2627. [CrossRef] [Medline]
- McGill M, Felton A, Global Partnership for Effective Diabetes Management. New global recommendations: a multidisciplinary approach to improving outcomes in diabetes. Prim Care Diabetes 2007 Feb;1(1):49-55. [CrossRef] [Medline]
- Sousa W, Vitorino P, Sousa A, Jardim T, Souza W, Moreira H, et al. PP.30.35. Journal of Hypertension 2015;33:e408 [FREE Full text] [CrossRef]
|CDSS: clinical decision support system|
|CHW: community health worker|
|CKD-EPI: Chronic Kidney Disease Epidemiology Collaboration|
|FHS: family health service|
|HDI: Human Development Index|
|NASF-AB: Núcleo Ampliado de Saúde da Família e Atenção Básica|
Edited by G Eysenbach; submitted 24.03.20; peer-reviewed by S Omboni, S Sarbadhikari; comments to author 25.07.20; revised version received 19.09.20; accepted 21.09.20; published 11.01.21Copyright
©Milena Soriano Marcolino, João Antonio Queiroz Oliveira, Christiane Corrêa Rodrigues Cimini, Junia Xavier Maia, Vânia Soares Oliveira Almeida Pinto, Thábata Queiroz Vivas Sá, Kaique Amancio, Lissandra Coelho, Leonardo Bonisson Ribeiro, Clareci Silva Cardoso, Antonio Luiz Ribeiro. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 11.01.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.