COVID-19 Mobile Apps: A Systematic Review of the Literature

Background A vast amount of mobile apps have been developed during the past few months in an attempt to “flatten the curve” of the increasing number of COVID-19 cases. Objective This systematic review aims to shed light into studies found in the scientific literature that have used and evaluated mobile apps for the prevention, management, treatment, or follow-up of COVID-19. Methods We searched the bibliographic databases Global Literature on Coronavirus Disease, PubMed, and Scopus to identify papers focusing on mobile apps for COVID-19 that show evidence of their real-life use and have been developed involving clinical professionals in their design or validation. Results Mobile apps have been implemented for training, information sharing, risk assessment, self-management of symptoms, contact tracing, home monitoring, and decision making, rapidly offering effective and usable tools for managing the COVID-19 pandemic. Conclusions Mobile apps are considered to be a valuable tool for citizens, health professionals, and decision makers in facing critical challenges imposed by the pandemic, such as reducing the burden on hospitals, providing access to credible information, tracking the symptoms and mental health of individuals, and discovering new predictors.


Study Selection
The string "(mobile health) OR (mhealth) OR (smartphone) OR (mobile phone) OR (mobile application) OR (mobile app) OR (app) AND (COVID- 19)," was used for a search within the title, abstract, and keywords of the manuscripts. Authors HK, AT, AK, FL, DGK, and IK independently screened the identified papers to minimize possible errors and bias in the selection process. Any disagreements were resolved by discussion between the authors to reach consensus. The authors first screened the abstracts of the candidate papers for inclusion and assessed their eligibility according to the defined inclusion and exclusion criteria. Moreover, the authors selected the final papers for inclusion after reading the full manuscripts of the eligible papers, as well as their references.
The Effective Public Health Practice Project (EPHPP) tool was adopted to assess the methodological quality of the included studies. The EPHPP tool is suitable for evaluating quantitative studies in a wide range of health-related topics, and it has demonstrated reliability [21]. The included studies were synthesized (by HK, AK, FL, and DGK) according to their target, mobile app main features, study design, number of enrolled participants and their age, follow-up duration, outcomes and whether these were positive or negative, and implications for clinical practice. This systematic review was conducted following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [22]. A completed PRISMA checklist is shown in Multimedia Appendix 1.

Statistical Analysis
Our last search in June 2020 returned 165 manuscripts from the PubMed database, 188 manuscripts from the Scopus database, and another 375 manuscripts from the WHO research database. All the retrieved records were imported in the Zotero bibliography management software (Center for History and New Media at George Mason University), which identified 252 duplicates. We screened the abstracts of the remaining 476 papers according to our inclusion and exclusion criteria, and 22 papers were found to be eligible. After reading the full text of the papers, the authors agreed to include 12 papers. The screening procedure along with reasons for excluding papers are shown in the PRISMA flow diagram in Figure 1.

Study Outcomes and Quality Assessment
On the basis of the EPHPP criteria for selection bias, design, confounders, blinding, data collection, and dropouts, we found the methodological quality to be moderate for 2 of the 12 (17%) studies [23,24] and weak for the remaining 10 (83%) studies (Table 1). Most studies were poorly rated because of their observational or cross-sectional nature, insufficient care in controlling for confounders, insufficient reporting on the validity and reliability of the tools used for data collection, and the absence of description on withdrawals and dropouts. The design of a randomized or controlled clinical trial was not reported in any of the reviewed studies.

Comparison
In the following sections, we provide a comparison of the included studies, in terms of intervention target and main features, study design, and outcome and clinical implications.

Intervention Target and Main Features
The reviewed papers varied in intervention target and main features ( Table 2). One paper discussed a COVID-19 tracking app informing users about risk assessment and offering relevant advice [28]. The paper did not provide further details about app features. Apps that offered education and information material were discussed in 3 papers. They either targeted patients or health care professionals. Patient education included hospital information (appointments, visiting hours, self-isolation, and general COVID-19 information [32]) and education related to SARS-CoV-2. For health care professionals, they included information about how to handle patients with COVID-19 [31,34]. One paper [25] described how the hospital information system was adapted to handle COVID-19 with the incorporation of relevant templates. An app for recording general health information was discussed in 1 paper [33], and apps for self-assessment focusing on symptoms were found in 1 paper [32]. Self-monitoring based on a symptom diary and health observation data were also reported in 4 papers [26,29,32,33].

Main features Intervention Studies
Risk assessment and advice Tracking app Kodali et al [28] Information for patients Education and information Timmers et al [32] Information for health care professionals Education and information Ros and Neuwirth [31] and Zamberg et al [34] Adaptation to pandemic Hospital information system Bourdon et al [23] General Health assessment Yamamoto et al [33] Self-monitoring Health assessment Bae et al [25], Medina et al [29], Timmers et al [32], and Yamamoto et al [33] Telephone outreach Home monitoring Medina et al [29] Vital signs with smart devices Home monitoring Bourdon et al [23] Internet of Things Home monitoring Drew et al [26] Demographics and health status Interactive map Timmers et al [32] Patient email to specific recipients Data sharing Yamamoto et al [33] Nurse outreach Data sharing Medina et al [29] Behavior during the pandemic Student mental health Ben Hassen et al [27] Progress of disease Prediction model Menni et al [30] Emergency eye care Teleophtalmology Huckins et al [24] Home monitoring was handled based on telephone outreach [29], smart vital sign monitoring devices [25], and Internet of Things [27]. An interactive map of user demographics and health status was discussed in 1 paper [32]. Two apps offered data sharing capabilities via email to specific recipients based on patient options [33], or as symptoms were worsening, the app sent messages to be assessed by nursing staff to provide appropriate advice [29]. One paper assessed student behavior to determine whether the mental health of participants changed due to the COVID-19 pandemic [24]. One paper used data collected by Drew et al [26] to develop a prediction model about disease progress as well as tracking the disease progress in real time [30]. Finally, 1 paper [23] reported on a smartphone app to manage emergency eye care.

Study Design and Structure of Research
Regarding the study design, almost all selected studies were proof of concept, observational studies, or both, aiming to evaluate the practical use, usability, or user satisfaction with the corresponding apps that were developed to support individuals, health care providers, and policy makers during the COVID-19 pandemic (Table 3).  [24] number of patients older than 60 years was small. The StudentLife app was used for smartphone mobile sensing. Ecological momentary assessments were used to assess depression and anxiety.
All reviews that were available publicly and Not reported 503 most relevant reviews were identified Observational study using descriptive statistics and thematic analysis on the mHealth a app Arogya Setu.
Kodali et al [28] posted in English by the based on the Google algorithm users until April 21, 2020, were included. The start date of app reviews collection was not reported.
Engagement with My-Care Companion app 25% (n=483) were older than 60 years, and COVID-19 patients enrolled by May 25, 2020: Observational cohort study carried out at the Cleveland Clinic, OH, US. It included a self-monitoring app for patient engagement and early intervention.
for longer than 14 days due to persistent symptoms.

Data analyzed had been collected between
Average age for tested positive, tested nega- Observational cohort study (based on the data collected at the ETZ b hospital), assessed the use of the app as well as its usability. Data were gathered for health care providers and policy makers.
Timmers et al [32] The active epidemiological investigation period was carried out from March 6-19, 2020, at Wakayama City Public Health Center. In this period 57 of 72 individuals (health observers) adopted the use of the app. By mid-May, the app had been used by more than 20,280 users and 400 facilities and organizations. These included companies, schools, hospitals, and local governments across Japan.
N/A d In the context of the active epidemiological investigation period (from March 6-19, 2020) at Wakayama City Public Health Center, 72 individuals who had close contact with a COVID-19 confirmed case were discovered. Among them, 57 had adopted the use of the health observation app.
Proof of concept and practical use study in a real-world setting. The study aimed to develop a PHR c -based COVID-19 symptom-tracking app to determine whether PHRs could be used for efficient health observation outside a traditional hospital setting. The practical aspects of health observations for COVID-19 using the smartphone or tablet app integrated with PHRs was demonstrated. Moreover, a usability evaluation of the app was carried out based on interviews with help desk managers of the app.
Yamamoto et al [33] The mHealth platform was used for 18  One study [24] used the corresponding mobile app (StudentLife app) as a tool for an ecological momentary assessment, aiming to assess depression and anxiety of college students to determine whether there was a change in behavior and mental health of participants in response to COVID-19.
The number of participants was mentioned in all selected studies. Some studies reported a relatively low numbers of participants using the corresponding app, whereas others a relatively large number of participants and users. The studies that reported the largest number of users were those described by Drew et al [26] (over 2 million users) and Menni et al [30] (2,450,569 users in the United Kingdom and 168,293 from the United States that had reported symptoms through the smartphone app). Studies that reported the lowest number of participants were the ones described by Bae et al [25] (12 patients and 24 medical staff) and Ben Hassen et al [27] (5 hospitalized patients and 5 doctors).
The age of participants was also reported in most of the studies except 3 [28,31,33]. The average age or age range for participants varied depending on the corresponding study.
Concerning the follow-up duration (study duration), 3 out of the 12 selected studies [23,25,27], did not report the study duration. Kodali et al [28] reported the end date but did not mention the start date of app use, Drew et al [26] reported only the start date of the app launch, Ros and Neuwirth [31] reported 3 days of app use, Yamamoto et al [33] reported 14 days, Zamberg et al [34] reported 18 days, Timmers et al [32] reported 20 days, and Huckins et al [24] reported 67 days of app use. Finally, Medina et al [29] mentioned that the engagement with the app reached 32%, whereas 25% of users continued under monitoring for longer than 14 days due to persistent symptoms.

Outcomes
Outcome assessments were varied across studies. An overview is shown in Table 4. • The ability of individuals to record health status on a daily basis was an important countermeasure against COVID-19.
• 72 health observers were identified who were in close contact with a confirmed case. Among them, 57 adopted the app, while 14 used telephone as a means for conducting investigations.
• The use of the app improved the efficiency and completeness of the investigation process for • Before the introduction of the app, phone interviews required more than 2 hours and four epidemiological officers for contact tracing.
After the introduction of the app, only one epidemiological officer was COVID-19 cases carried out by epidemiological needed to perform health observations. officers.

•
The visualization of health observation data improved the investigation efficiency and comprehensiveness.
Zamberg et al [34] • Using the mHealth solution as a communication channel turned out to be effective within the organization for dissemination purposes during the • Error correction, improved data collection quality, and user privacy should be considered in mHealth apps.
• Mixed evidence about the use of the app but mainly optimistic

•
Steps must be taken to ensure the reliability of the information provided by users. Therefore, predicting multiple verification of data entered by users could be critical.

Huckins et al [24]
• Mobile apps can be effectively used for tracking the mental health of college students.
• With the rise of news relevant to COVID-19, college students spent more time seated, had fewer visits, and showed increases in anxiety and depression.

•
The authors did not observe a return to baseline over the break, although they observed decreases in stress and depression that paralleled the typical drop after the final examination, suggesting some resilience in the face of COVID-19.

Drew et al [26]
• With the participation of groups with underrepresented populations, the study aimed to encourage enrollment of individuals from populations that

•
The app captures the dynamics of COVID-19 onset days before traditional measures such as positive tests, hospitalizations, or mortality.

•
The collection emphasizes the potential usefulness of symptom monitoring in real time to help guide the allocation of resources for testing have traditionally been difficult to recruit. and treatment, as well as advising for tightening or loosening appropri-• The study could capture correlations based on individual variations over time, a remarkable advantage ate measures in specific areas.
over repetitive cross-sectional surveys that introduced significant variation between individuals.
Positive/negative outcomes Primary outcomes Paper

•
Mobile engagement platforms have the potential to reduce the need for caregiver communication for patients whose symptoms are mild or persistent, freeing up the health care professionals to focus on patients who need it more.
• Mobile and home-based interventions were feasible for a wide range of conditions with a related risk of poor outcome from COVID-19.

•
Approximately 10% of the patients in active monitoring presented symptoms such as shortness of breath that required escalation to a virtual provider. The median time to escalation ranged between 7 and 8 days.

•
Patients with a pulse oximeter at home escalated a few days earlier due to reduced oxygen saturation measurements before subjective complaints of dyspnea. • 2% of patients in active supervision were eventually admitted, and 3% were readmitted for persistent COVID-19 symptoms or due to complications of other underlying diseases. • 9 patients monitored at home died, either due to complications related to COVID-19 or complications of another underlying disease.

Medina et al [29]
• Physiological assessments of olfactory and taste function or nucleotide-based testing for SARS-CoV-2 could not be replaced by self-reporting.

•
The authors did not know if anosmia was acquired before or after other COVID-19 symptoms, or during or after the illness.
• Besides more established symptoms such as high fever and a persistent cough, loss of smell and taste were possible prognostic factors for COVID-19.

•
A combination of symptoms such as anosmia, fatigue, persistent cough, and loss of appetite together could identify individuals with COVID-19.
Menni et al [30] Advantage over medical videos: • It allowed the user to live the experience of seeing through a first-person view to learn through the eyes of the expert.

•
The ability for a health care professional to instantly download locally (in a smart phone) material that can be accessed at any time in real time before, during, or after patient care interactions.

Ross and Neuwirth [31]
• During periods of pandemics and disasters, automated exchange of information between health care institutions plays an important role in dealing more efficiently with the problem at hand.

•
Mobile app: usefulness showed the highest score, followed by satisfaction and perceived ease of use.

•
Wearable vital sign monitoring perceived usefulness scored the highest, followed by perceived ease of use and satisfaction. • For carers, there was an overall satisfaction score of 4.10/5.

•
Vital signs had to be measured by the patients themselves.

•
Video communication between patients and doctors was added.
• Patients and doctors alike accepted the home hospitalization system very well.

•
Average delay of 4.12 days between the onset of symptoms and advice, and <1 day for emergency episodes.

•
There was 96% sensitivity and 95% specificity for the correct evaluation of the indication of a physical consultation and only 1.0% misdiagnoses.
• Allowed doctors and patients to maintain social distance, avoiding three or four physical trips per person.
In most of the cases, app users were satisfied with the educational and risk assessment information provided in the mobile apps (both citizens [32] and health personnel [31,34]). They felt reassured and informed. However, there were also concerns [28] about the measures that would need to be taken to ensure the reliability of information provided to users.
In addition, 3 studies [26,30,32] showed the feasibility of the implementation and use of COVID-19 apps to support education, self-management, and symptom tracking through diaries. Symptom tracker apps had even identified potential predictors of COVID-19, such as loss of smell and taste [30]. To this direction, mobile app sensing revealed that, during the COVID-19 pandemic, individuals were more sedentary, visited fewer locations, and showed increases in anxiety and depression [24], which is in line with the "Stay safe, stay home" policies mandated by local and national governments. However, there were concerns that cross-verification of data entered by the users could be crucial [28] and that the self-reported nature of the data collected by these apps could not replace physiological assessments and clinical examinations [30].
Additionally, mobiles apps were used for home monitoring, as reported in 4 studies [23,25,27,29], with satisfactory usability for both patients and health care professionals. These apps were focused on the management of patients who are at high risk, the moderation of exposure risk for health care workers, and the reductions of community spread through appropriate education on home-based care for individuals who are exposed or infected.
Based on the data reported in the reviewed papers, interactive maps and dashboards could be created for the quick visualization of the status summary of patients with COVID-19 and disease spread to be used by policy makers and health care providers for decision making at regional levels [26,32,33]. In addition, the dynamics of incidence days could be effectively captured, guiding allocation of resources for testing, treatment, and lockdown recommendations [26]. Mobile apps could significantly improve the efficiency and completeness of contact tracing workflows [33]. Our main finding is that, despite that the current research evidence is fragmented and requires greater methodological rigor, mobile apps have been found to benefit citizens, health professionals, and decision makers in facing the COVID-19 pandemic. In particular, mobile apps can help in solving several COVID-19-related challenges by increasing the reach of reliable information to both citizens and health professionals, decreasing misinformation and confusion, tracking symptoms and mental health of citizens, home monitoring and isolation, discovering new predictors, optimizing health care resource allocation, and reducing the burden of hospitals.

Main Findings
The participants in the studies were mainly young and middle-aged adults. Further studies are needed that will involve older participants, who are in greater risk of developing serious complications due to COVID-19. Understanding the needs of older individuals in the COVID-19 pandemic period would be the first necessary step toward designing and developing mobile apps to encourage their physical and mental well-being [17].
Our review, in contrast to other reviews that have not examined the evaluation of COVID-19 mobile apps in pragmatic studies [8], identified that the majority of included studies were not of high methodological quality, mainly because of their observational nature. This could be justified by the fact that the COVID-19 pandemic crisis generated an international appeal for fast response and rapid development of digital health tools by the research community, which might have inevitably led to the publication of early results by observational studies. This result can be seen as complementary to other reviews [8] that report that many of those apps are of high quality, offering many functionalities and advanced user experience. Longitudinal studies with rigorous design such as randomized controlled trials are now required to systematically assess COVID-19 mobile apps and provide strong evidence of their value. However, ethical implications might arise due to possible conflicts between liberty and privacy, equity, fairness, and justice [35][36][37]. In this direction, health outcomes that have scarcely been used so far, such as infection rate and quality of life, could be used as primary end points.

Implications for Clinical Practice
The implications for clinical practice for each one of the discussed works is shown in detail in Textbox 1.

Timmers et al [32]
• eHealth mobile apps for COVID-19 that support several functionalities such as tracking of symptoms and provision of accurate and timely information as well as self-assessment can be implemented in a short time to be used by individuals.
• Those apps provide valuable information to both governments and health care providers, since they support monitoring of patient health status and provide summary statistics regarding the progress of health.
• Such apps could be used in future outbreaks of other viruses to support all involved stakeholders.

Yamamoto et al [33]
• Such apps could be used in future outbreaks of other viruses to support all involved stakeholders.

Zamberg et al [34]
• Mobile health (mHealth) apps could help solve some of the COVID-19 challenges by providing more accurate and timely information to health care professionals. This benefit is sourced by the centralized management and storage of main, up-to-date, validated, and easily accessed information in one platform.
• mHealth apps used in health care organizations as communication tools of validated information should be assessed in the context of clinical studies, with regards to their impact on clinical outcomes.

Kodali et al [28]
• According to users, the app should be enhanced with additional functionalities including tracking of location, provision of up-to-date information on COVID-19 as well as information on areas with high/medium/low epidemiological burden, and deployment in nonmobile platforms.

Huckins et al [24]
• According to the findings, since the beginning of the COVID-19 pandemic, there has been an extended negative impact both on the physical health of individuals, resulting in an increased number of deaths, and in their mental health, resulting in changes in their behavior.

Drew et al [26]
• The research of a broader range of potential risk factors for COVID-19 results will be largely supported by the use of the app within several large epidemiology cohorts for which there are a large amount of data on lifestyle, diet and health factors, and genetic information.
• It will be useful to deploy the tool in several clinical studies, in centralized actions related to biobanking, and in health care worker monitoring programs.

Medina et al [29]
• The intervention impacts several organizational matters and provides answers regarding the following questions related to COVID-19: • Which situations should we focus on to provide effective care to prevent patient admission to intensive care unit and mechanical support?
• How can those situations be better supported by increased patient self-reporting?
• Can we predict, based on time data, future inpatient demand?
• How can we manage not having conflicts between patient choice and availability of treatment?

Menni et al [30]
• Routine screening for COVID-19 could also include loss of sense of taste and smell. These two symptoms should also be included in the related symptom list provided by the World Health Organization.

Ros and Neuwirth [31]
• The implementation and deployment of the described digital tutorials, as an effective and swift global public health educational tool, help alleviate the burden that hospitals, health care professionals/responders, and patients face due to the COVID-19 pandemic.

Bae et al [25]
• Episode triage, timely diagnosis, isolation of patients, and their treatment can be largely supported by telemedicine solutions.

Ben Hassen et al [27]
• Mobile apps can be used by patients, their caregivers, and health care professionals to better monitor and manage patient health status in the context of patient hospitalization. Moreover, such apps are cost-effective, reliable, and safe, providing important economic benefits to hospitals. Overall, they are accepted to a great extent by patients and individuals.

Bourdon et al [23]
• The use of telemedicine tools for consultation purposes improved the access to health care services for patients with ophthalmological problems. At the same time, such tools preserved social distancing and sanitary measures.
• Such tools can be used in emergency situations by ophthalmological patients who have limited access to specialized care.
As reported in the textbox, eHealth solutions can be implemented rapidly and can offer essential tools in supporting the COVID-19 pandemic for all stakeholders including citizens, health care providers, policy makers, and governments. The impact of the COVID-19 pandemic goes beyond the illness and deaths that are directly associated with the SARS-CoV-2 pathogen and toward an expanded scope encompassing mental health and several behavioral changes.
However, there are still several areas to explore, such as better geolocation tracking; timely COVID-19 updates; deployment on nonsmartphone platforms; better incidence visualization and prediction; investigation of a much broader range of putative risk factors for COVID-19 outcomes (based on other longitudinal data on lifestyle, diet, genetics, etc); and the integration with clinical systems, digital health literacy, and engagement with mobile apps.
Apps such as the ones presented in these studies can give the research community the opportunity to monitor long term effects of COVID-19; answer questions such as the number of people truly affected, why some people get sicker than others, and how long people can stay immune to the disease; and perform better resource allocation of medical equipment.
In case of future disease outbreaks, the mobile apps already developed for COVID-19 can be valuable tools ready to support people, health care providers, and policy makers. This study can provide a guide for future developers and researchers regarding the current methodological gaps and challenges that need to be addressed to develop well-designed and evaluated apps for similar future circumstances.

Limitations
We used terms related to mobile apps, mHealth, and mobile phones for our literature search according to the review objectives and did not use related terms such as "telehealth," which might have resulted in an inadvertent omission of studies. Our literature search was conducted in a limited number of bibliographic databases (Pubmed, Scopus, and the WHO's COVID-19 database), which nevertheless have been largely used worldwide. The gray literature was not explored. The interrater reliability between the authors was not assessed. A meta-analysis was not possible due to the heterogeneity of the included studies. For many cases, the quality of the study might not reflect the quality or the effectiveness of the developed mobile app. As such, useful and effective apps might have not been included in this review study due to the limited quality of the related studies.