This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The coronavirus disease (COVID-19) has become an urgent and serious global public health crisis. Community engagement is the first line of defense in the fight against infectious diseases, and general practitioners (GPs) play an important role in it. GPs are facing unique challenges from disasters and pandemics in delivering health care. However, there is still no suitable mobile management system that can help GPs collect data, dynamically assess risks, and effectively triage or follow-up with patients with COVID-19.
The aim of this study is to design, develop, and deploy a mobile-based decision support system for COVID-19 (DDC19) to assist GPs in collecting data, assessing risk, triaging, managing, and following up with patients during the COVID-19 outbreak.
Based on the actual scenarios and the process of patients using health care, we analyzed the key issues that need to be solved and designed the main business flowchart of DDC19. We then constructed a COVID-19 dynamic risk stratification model with high recall and clinical interpretability, which was based on a multiclass logistic regression algorithm. Finally, through a 10-fold cross-validation to quantitatively evaluate the risk stratification ability of the model, a total of 2243 clinical data consisting of 36 dimension clinical features from fever clinics were used for training and evaluation of the model.
DDC19 is composed of three parts: mobile terminal apps for the patient-end and GP-end, and the database system. All mobile terminal devices were wirelessly connected to the back end data center to implement request sending and data transmission. We used low risk, moderate risk, and high risk as labels, and adopted a 10-fold cross-validation method to evaluate and test the COVID-19 dynamic risk stratification model in different scenarios (different dimensions of personal clinical data accessible at an earlier stage). The data set dimensions were (2243, 15) when only using the data of patients’ demographic information, clinical symptoms, and contact history; (2243, 35) when the results of blood tests were added; and (2243, 36) after obtaining the computed tomography imaging results of the patient. The average value of the three classification results of the macro–area under the curve were all above 0.71 in each scenario.
DCC19 is a mobile decision support system designed and developed to assist GPs in providing dynamic risk assessments for patients with suspected COVID-19 during the outbreak, and the model had a good ability to predict risk levels in any scenario it covered.
In December 2019, cases of pneumonia with unknown cause, which was designated the coronavirus disease (COVID-19) in February by the World Health Organization (WHO), were reported in Wuhan, Hubei Province, China [
As a new infectious disease in humans caused by severe acute respiratory syndrome coronavirus 2 (SARS-COV-2), COVID-19 is characterized by respiratory symptoms and human-to-human transmission [
Community engagement is the first line of defense for effective prevention and control of infectious diseases [
Disasters and pandemics pose unique challenges to health care delivery [
Telemedicine has a critical role in emergency responses and is an ideal model for managing infectious diseases [
Until now, researchers have developed many different forms of telemedicine systems to meet the needs of fighting against the epidemic. Jin et al [
These models and systems have considered the main problems from different perspectives during the COVID-19 epidemic and have been deployed and published in a short time, which has played a significant role in fighting the epidemic. However, to the best of our knowledge, there is still no suitable mobile management system that can help GPs realize the automatic collection of data, dynamic risk assessment, and effective triaging and follow-ups with patients with COVID-19, as well as the effective reduction of the pressure on large designated general hospitals.
In this study, by integrating doctor experience, clinical guidelines, and retrospective data, we designed and developed a dynamic risk assessment decision support system for COVID-19 (DDC19) to assist GPs in data collection, dynamic risk assessment, triage management, and follow-ups during the outbreak of COVID-19. The DDC19 is designed to build a free mobile app that can cover all the different situations encountered by residents and GPs, and GPs can use it for dynamic continuity management. We describe our experiences, lessons learned, and recommendations for the design and implementation of telemedicine tools in future health emergencies.
To achieve the early assessment and triage of patients with COVID-19 and ease the pressure of shortages in medical resources, the following key issues need to be resolved: how to fully grasp and effectively manage the residents’ status in real time without increasing the GP's working burden and, without omitting potential patients with COVID-19, how to effectively use medical knowledge and risk stratification models to achieve effective evaluation and classification, as well as the patients’ scientific stratification.
Accordingly, based on the principle of the four early approaches (early detection, early reporting, early isolation, and early treatment) and the actual scenes and process of patients using health care, DDC19 was designed to help GPs manage their patients who had a fever or respiratory symptoms, or suspected infection with SARS-CoV-2 during the outbreak of COVID-19. Several scenarios were involved and are demonstrated in
The covering scenes of patients using health care during the outbreak of COVID-19. COVID-19: coronavirus disease; CT: computed tomography; GP: general practitioner.
To make full use of the previously mentioned online-offline combined triage mode during the COVID-19 outbreak, we deeply integrated GP’s experience and suggestions, and designed the business process of the DDC19 system in detail. In the first step, we reviewed the COVID-19 guidelines [
As shown in
The main business flowchart of DDC19. CT: computed tomography; GP: general practitioner; OCR: optical character recognition.
Considering the characteristics of COVID-19 such as unobvious symptom specificity and long incubation period, we constructed a COVID-19 dynamic risk stratification model with high recall and clinical interpretability. Based on a multiclass logistic regression algorithm, it integrates the retrospective clinical data analysis results of patients, doctors’ experiences, and clinical guidelines.
To meet the risk assessment requirements of COVID-19 at the system level for multiple levels and multiple scenarios, this model was based on patients’ data from the fever clinic during the COVID-19 epidemic period and patients’ risk levels provided by GPs according to COVID-19 diagnosis and treatment guidelines [
Within 14 days before the onset of the disease, the patient has a travel or residence history in the high-risk regions or countries.
Within 14 days before the onset of the disease, the patient has a history of contact with those infected with severe acute respiratory syndrome coronavirus 2 (those with a positive nucleic acid testing result).
Within 14 days before the onset of the disease, the patient had direct contact with patients with fever or respiratory symptoms in high-risk regions or countries.
Disease clustering (2 or more cases with fever or respiratory symptoms occur at such places as homes, offices, and school classrooms, within 2 weeks)
Fever or respiratory symptoms
The white blood cell counts in the early stage of the disease is normal or decreased, or the lymphocyte count decreases over time.
Computed tomography imaging features of the coronavirus disease
The construction process of the dynamic risk stratification model.
The data elements in the dynamic risk stratification model.
Main category, data element | Description |
|
|
|
|||
|
Patient ID | Unique patient identifier |
|
|
Gender | Patient’s gender identity |
|
|
Age | Patient’s age |
|
|
|||
|
Fever | Normal: temprature≤37.2℃; low fever: temperature between 37.2℃ and 38.5℃; High fever: temperature≥38.5℃ |
|
|
Cough | Cough or dry cough |
|
|
Sputum production | N/Aa |
|
|
Fatigue | N/A |
|
|
Breathing | Shortness of breath, anhelation, polypnea, etc |
|
|
Chest uncomfortable | Chest pain or chest distress |
|
|
Pharyngalgia | Pharyngalgia |
|
|
Headache | Headache or dizziness |
|
|
Chills | Fear of cold |
|
|
Soreness | Body aches, joint pain, myalgia |
|
|
Stuffy nose | Stuffy nose or runny nose |
|
|
Gastrointestinal reactions | Feeling sick, vomiting, abdominal pain, diarrhea, etc |
|
|
|
||
|
Contact history | Have a COVID-19b contact history |
|
|
|
||
|
CTc | Lung CT shows viral pneumonia |
|
|
|||
|
WBCd | White blood cell count (10E9/L) |
|
|
GRANe | Neutrophil count (10E9/L) |
|
|
LYMf | Lymphocyte count (10E9/L) |
|
|
RBCg | Red blood cell count (10E12/L) |
|
|
HGBh | Hemoglobin concentration (g/L) |
|
|
HCTi | Hematocrit (%) |
|
|
MCVj | Mean corpuscular volume (fl) |
|
|
MCHk | Mean hemoglobin content (pg) |
|
|
MCHCl | Mean corpuscular hemoglobin concentration (g/L) |
|
|
RDWm | Red blood cell distribution width (%) |
|
|
PLTn | Blood platelet count (10E9/L) |
|
|
MPVo | Mean platelet volume (fl) |
|
|
PCTp | Platelet hematocrit (%) |
|
|
PDWq | Platelet distribution width (10 [GSDr]) |
|
|
MOs | Mononuclear cell count (10E9/L) |
|
|
EOt | Eosinophil count (10E9/L) |
|
|
BAu | Basophil count (10E9/L) |
|
|
NRBCv | Percentage of nucleated red blood cells |
|
|
IGw | Immature granulocyte percentage (%) |
|
|
CRPHx | C-reactive protein (mg/L) |
|
aNot applicable.
bCOVID-19: coronavirus disease.
cCT: computed tomography.
dWBC: white blood cell.
eGRAN: granulocytes.
fLYM: lymphocyte.
gRBC: red blood cell.
hHGB: hemoglobin.
iHCT: hematocrit.
jMCV: mean corpuscular volume.
kMHC: mean hemoglobin content.
lMCHC: mean corpuscular hemoglobin concentration.
mRDW: red blood cell distribution width.
nPLT: platelet.
oMPV: mean platelet volume.
pPCT: platelet hematocrit.
qPDW: platelet distribution width.
rGSD: geometric standard deviation.
sMO: mononuclear.
tEO: eosinophil.
uBA: basophil.
vNRBC: nucleated red blood cells.
wIG: immature granulocyte.
xCRPH: C-reactive protein.
After the optical character recognition (OCR) module and natural language processing (NLP) module recognize and preprocess the image and text data, the structured patient data is extracted from the app back end database to the data preprocessing module for subsequent data analysis; the detailed data elements are shown in
In this formula,
The output
To measure the difference between the model prediction classification result
For m training samples, the loss function is expressed as follows:
Based on the retrospective data of m patients, by minimizing the loss function
Where i is the i-th category. For the input data of the newly visited patient,
To prove the effectiveness of the previously mentioned COVID-19 dynamic risk stratification model, as well as the feasibility and practicability of the early assessment and triage system, the research team developed the system prototype and put it into practical operation and application.
As shown in
System architecture diagram. API: application programming interface; FHIR: Fast Healthcare Interoperability Resources; HL7: Health Level 7; RESTFUL: representational state transfer.
The mobile app end builds user interface based on the progressive Vue framework. The server end implements data and service interaction with the front end by the representational state transfer application programming interface specification. It is more concise and lighter, both for the processing of uniform resource locators and the encoding of the payload [
For the patient-end (
The screenshots of DDC19’s patient mobile terminal app.
For the GP-end (
The screenshots of DDC19’s doctor mobile terminal app.
For the purpose of testing and verifying the dynamic risk stratification model of DDC19, 2243 patients’ information were collected from the fever clinic of the first affiliated hospital of Zhejiang University from January 19 to March 11, 2020. The data includes the patients’ basic information (gender and age), chief complaint, medical history, physical examination, laboratory tests, and the lung CT image examination reports. The first affiliated hospital of Zhejiang University is a class A hospital with 2500 beds. In 2019, the number of outpatient and emergency services reached 5 million and 243,300 were discharged.
The patient's medical record number is used as the unique identifier, and the data corresponding to the earliest outpatient visit record of each patient within the range from January 19, 2020, to March 11, 2020, is taken. In terms of data preprocessing, except for age, which is used as a continuous variable, the other data are classified as categorical variables. The imaging examination results were divided into clearly marked “viral pneumonia,” other lung function categories, and no abnormalities. According to the first hospital’s reference threshold, the results of laboratory tests were classified as “lower/normal/higher” or “normal/higher.” Age was standardized, and other missing elements were filled with 0.
In our study, according to the risk assessment model, patients were divided into three groups: high-risk group, moderate-risk group, and low-risk group. Out of the 2243 patients, 628 (28.00%) were in the low-risk group, 1447 (64.51%) were in the moderate-risk group, and 168 (7.49%) were in the high-risk group. Among them, 17 patients were clinically diagnosed with COVID-19; 16 patients were in the high-risk group, and 1 patient was in the moderate-risk group (see details in
To ensure the accuracy of risk stratification and to avoid the model overfitting, we used the three categories of low risk, moderate risk, and high risk as labels, and adopted a 10-fold cross-validation method to evaluate and test the model in different scenarios (different dimensions of personal clinical data accessible at the early stage). When we only used the data of patients’ demographic information, clinical symptoms, and contact history, the data set dimensions were (2243, 15). When the results of blood tests were added, the data set dimensions were (2243, 35). After obtaining the CT imaging results of the patient, the data set dimensions were (2243, 36; see details in
The ROC curve of the dynamic risk stratification model. ROC: receiver operating characteristic.
As can be seen from the ROC curve and its corresponding AUC value, even if only the information of the patient's epidemiology contact history and clinical symptoms were used, the average value of the three classification results of macro-AUC were all above 0.71. When the data of laboratory tests and imaging were added, the macro-AUC increased to above 0.97. Therefore, the model has a good prediction ability for the above three scenarios. The detailed evaluation indicators of the model are shown in
The indicators of the model.
Variables | Situation 1 | Situation 2 | Situation 3 | ||||||||
|
Precision | Recall | F1 score | Precision | Recall | F1 score | Precision | Recall | F1 score | ||
Class 1 | 0.380 | 0.576 | 0.456 | 0.947 | 0.956 | 0.951 | 0.949 | 0.956 | 0.952 | ||
Class 2 | 0.750 | 0.552 | 0.634 | 0.976 | 0.956 | 0.966 | 0.980 | 0.957 | 0.968 | ||
Class 3 | 0.750 | 0.947 | 0.831 | 0.841 | 0.941 | 0.885 | 0.850 | 0.982 | 0.909 | ||
Accuracy | 0.588 | 0.588 | 0.588 | 0.955 | 0.955 | 0.955 | 0.959 | 0.959 | 0.959 | ||
Macroaverage | 0.627 | 0.692 | 0.640 | 0.921 | 0.951 | 0.934 | 0.926 | 0.965 | 0.943 | ||
Weighted average | 0.646 | 0.588 | 0.599 | 0.958 | 0.955 | 0.956 | 0.961 | 0.959 | 0.959 |
This paper describes a dynamic risk assessment decision support system, which has been used by many Chinese GPs in Zhejiang Province during the COVID-19 outbreak. The DDC19 was designed for GPs working in different situations such as online consultation, assessment evaluation, and triaging in different offline scenarios (community, airport, train station, fever clinic, etc) and following up with suspected patients and discharged patients. It fills in the gap of traditional health care and helps GPs effectively manage residents with different statuses. For patients in DDC19, they can use it to record health data, obtain real time results of risk assessment, and communicate with their GPs without in-person visits; for GPs, they can intuitively grasp their patients’ conditions and provide online advice and interventions in real time. DDC19 contributes to the effective triaging of patients, relieves the pressure of offline clinics of designated hospitals to a certain extent, and reduces cross-infection in the hospital during the COVID-19 outbreak.
With the worldwide spread of COVID-19 and the shortage of medical resources, achieving scientific assessment and effectively triaging the patients in different states is the key to control it. Under the actual clinical condition, GPs need to comprehensively evaluate patients offline without specific symptoms based on their epidemiological contact history, symptoms, laboratory, and imaging findings, and advise in conjunction with the guideline’s recommendations. Although it can assess patients, distinguish them with different risk levels, and find patients who are at high risk, it has a number of inherent defects that cannot meet the need for efficient prevention and control of COVID-19. At present, there is a lack of mobile medical information systems to meet the needs of patient classification, and it is infeasible to truly complete the triage of patients who are potentially infected with SARS-CoV-2 and other patients with similar symptoms before the outpatient clinic of the designated hospital. Researchers at the University of California, San Diego built a number of COVID-19–related tools to support physician’s work based on the EHR system [
In the traditional model, clinicians need to obtain patient’s clinical information within a short time by in-person visits. Due to the limitations of time and environment, it may be arduous to assess the patients’ risk for clinicians. In our study, DDC19 can dynamically obtain patients' clinical information by self-report or upload health information, assess their risk level by the dynamic risk stratification logistic regression model, and assist GPs to diagnose and provide recommendations. Therefore, DDC19 helps GPs with online triaging, reduces the pressure of offline clinic, and lowers the risk of cross-infection in the hospital. On the coverage of patient clinical information, our system has been further expanded through questionnaires and inspection reports. The multiclass logistic regression model constructed by the retrospective data (
DDC19 is also a new method for using clinical data, which can respond to emergencies conveniently and quickly. The existing COVID-19–related clinical analysis methods focus on establishing new hypotheses or searching for new research evidence. It is difficult to translate into a direct impact on accurate control and rapid response to the epidemic in a short time. Therefore, we built a complete mobile information system that integrates the workflow in the health care system. On the other hand, although there is a lot of research on informationization and AI-related to COVID-19 at the moment [
This study has some limitations that must be addressed in the next steps of development. First, the complete clinical situation should be considered during the establishment of the dynamic risk assessment model, including the severity of symptoms and history of underlying chronic diseases, but the early retrospective data of fever clinics do not contain these data elements. With further accumulation of relevant clinical data, our system can attempt earlier assessment and triage support methods, and we can also discuss and explore the impact of dynamic changes in patients' clinical information on their COVID-19 risk stratification. Second, as the system is still in the deployment and app stage, the relevant data of patients in the system and in-hospital visits cannot be obtained in a timely manner, so the clinical effects produced by the actual app of the system cannot be evaluated in a timely manner.
DCC19 is a mobile decision support system designed and developed to assist GPs in providing dynamic risk assessments for potential patients during the COVID-19 outbreak. It collects potential patients’ health information by mobile apps and data transmission mechanisms in different situations, assesses their risk levels through a dynamic risk stratification logistic regression model, and helps GPs manage patients and make further clinical decisions.
The questionnaire of health information.
Statistical distribution of patient data characteristics.
artificial intelligence
area under the curve
coronavirus disease
credible interval
C-reactive protein
computed tomography
electronic health record
general practitioner
natural language processing
optical character recognition
receiver operating characteristic
severe acute respiratory syndrome coronavirus 2
World Health Organization
This work was supported by the Major Scientific Project of Zhejiang Lab (2018DG0ZX01), the National Scientific and Technological Major Project of China (2018ZX10715014), “the Fundamental Research Funds for the Central Universities,” and the National Natural Science Foundation of China (81971982, 81771936, 81801796).
This work is based on the cross-cooperation of medical and bioinformatics researchers. JL and JR contributed equally in this work, including conceiving the original idea, designing the whole research process, and reviewing the manuscript. YL and ZW contributed to the design of the research process, data analysis, data interpretation, and wrote the first version of the manuscript. YT, MZ, TZ, and YQ contributed to the administration of the project, data analysis, and data interpretation. KY and YZ collected and cleaned the data. All the authors contributed to the interpretation of the results and the final manuscript.
None declared.