This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The COVID-19 pandemic has significantly impacted mental health and well-being. Mobile mental health apps can be scalable and useful tools in large-scale disaster responses and are particularly promising for reaching vulnerable populations. COVID Coach is a free, evidence-informed mobile app designed specifically to provide tools and resources for addressing COVID-19–related stress.
The purpose of this study was to characterize the overall usage of COVID Coach, explore retention and return usage, and assess whether the app was reaching individuals who may benefit from mental health resources.
Anonymous usage data collected from COVID Coach between May 1, 2020, through October 31, 2020, were extracted and analyzed for this study. The sample included 49,287 unique user codes and 3,368,931 in-app events.
Usage of interactive tools for coping and stress management comprised the majority of key app events (n=325,691, 70.4%), and the majority of app users tried a tool for managing stress (n=28,009, 58.8%). COVID Coach was utilized for ≤3 days by 80.9% (n=34,611) of the sample whose first day of app use occurred within the 6-month observation window. Usage of the key content in COVID Coach predicted returning to the app for a second day. Among those who tried at least one coping tool on their first day of app use, 57.2% (n=11,444) returned for a second visit; whereas only 46.3% (n=10,546) of those who did not try a tool returned (
As the mental health impacts of the pandemic continue to be widespread and increasing, digital health resources, such as apps like COVID Coach, are a scalable way to provide evidence-informed tools and resources. Future research is needed to better understand for whom and under what conditions the app is most helpful and how to increase and sustain engagement.
In the United States, the COVID-19 pandemic has led to over 500,000 deaths, millions of job losses, and disruption of nearly every aspect of daily life. COVID-19 has also negatively impacted mental health and well-being globally [
Several studies now indicate that an unprecedented mental health crisis is underway. In a poll conducted by Harris [
Digital mental health options are needed to help address the mental health effects of COVID-19 as well as the secondary impacts of the pandemic, such as fear of contracting the virus, financial stress related to job loss, loss of childcare, or the need to balance work with remote education. Mobile mental health apps are a promising strategy for addressing mental health impacts of the pandemic because of their potential scalability, reach, and utility, particularly during a time when in-person care may not be accessible due to social distancing and safety regulations. High-quality, accessible, and sustainable apps have been identified as part of an integrated “blueprint” for digital mental health services during the pandemic [
Apps are a particularly appealing medium because of their potential reach. Individuals rarely turn off mobile devices [
In response to the anticipated mental health impact of the COVID-19 pandemic, and as part of the Veterans Affairs’ (VA) “Fourth Mission” to help during times of national emergencies and support public health, the National Center for PTSD created COVID Coach (
COVID Coach is based upon the model of the empirically supported PTSD Coach app [
Mobile mental health apps can be useful tools in large-scale disaster responses [
This study utilized anonymous mobile analytics data to characterize the overall usage of an app designed specifically to provide tools and resources for addressing COVID-19–related stress, explore retention and return usage, and assess whether the app was reaching individuals that may benefit from mental health resources. Three key aims guided the study: (1) describe general usage trends between May 1, 2020, and October 31, 2020 (a key period of time during the pandemic), and identify how frequently specific types of key app content were used (ie, coping tools, psychoeducation, self-assessments, and accessing resources); (2) explore usage patterns, with a particular focus on understanding how usage of key content on the first day of use may be related to return use and retention; and (3) characterize baseline mental health and well-being among COVID Coach users.
COVID Coach, available for Android [
COVID Coach collects anonymous information about app use for the purposes of quality improvement. Fully nonidentifying, anonymous, and encrypted event sequences were stored using JavaScript Object Notation (JSON) format on a remote GovCloud server that meets VA security and privacy requirements. Data are accessible from VA App Connect software, which has been approved for use under the VA’s Technical Reference Model [
For the purpose of this study, mobile analytics data with timestamps between May 1, 2020, and October 31, 2020, were extracted from the research server on November 4, 2020. Between May 1 and October 31, 3,368,931 in-app related events were captured (Android: n=847,260; iOS: n=2,521,612) across 49,297 unique install codes (Android: n=12,938; iOS: n=36,359).
Daily active users and monthly active users were measured by the total number of app users that used COVID Coach on a given day or at least once within a given month. Overall, frequencies for key content usage were computed for each of the four key sections in the app:
Four assessments are available within the
The Warwick-Edinburgh Mental Well-Being Scale (WEMWBS) [
The Generalized Anxiety Disorder-7 (GAD-7) [
The Patient Health Questionnaire-9 (PHQ-9) [
The Posttraumatic Stress Disorder Checklist (PCL-5) [
SQLPro Studio (Hankinsoft Development, Inc) was used for all data preprocessing and extraction. SAS University Edition (SAS Institute) software in conjunction with Oracle’s VirtualBox were used for all data analyses. We calculated descriptive statistics for key content usage, retention, and baseline levels of mental health symptom severity and levels of well-being. Chi-square analyses were conducted to understand differences in returning to the app for a second day of use based on key content usage on the first day of app use and baseline mental health symptoms. We ran separate chi-square analyses for each predictor. Independent samples
The app was released at the end of April 2020, and as of October 31, 2020, it has been downloaded 143,097 times. It is highly rated on both the Apple App Store (4.8 out 5 stars) and the Google Play Store (4.7 out of 5 stars). Users had the opportunity to provide written reviews along with star ratings. The majority of written reviews were overwhelmingly positive, with comments such as “Beautifully calming…,” “a necessity for our new normal,” “one of the best free apps I’ve found,” and “this is amazing… it has all you may need… mood trackers, resources, meditation-not too frilly, just important.” Notably, due to Google’s restrictions on mobile apps related to COVID-19 (including hiding certain results for apps among searches containing “COVID”), COVID Coach has been installed at a ratio of over 3:1 for iOS compared to Android mobile devices.
The number of daily active users spiked in May 2020 (mean 1205.77, SD 615.70), shortly after the app’s release. The number of daily active users has leveled off but remained stable with average daily active users of 778.67 (SD 161.16), 752.03 (SD 152.07), 712.71 (SD 141.85), 682.83 (SD 150.83), and 611.35 (SD 128.60), respectively, during the months of June, July, August, September, and October 2020 (
Daily active COVID Coach users, from May 1, 2020, through October 31, 2020.
Within the observation window (May 1, 2020, through October 31, 2020), there were 49,297 unique app users and 462,651 app events associated with the four key content areas (
Of the four key sections of the app, the
Overall key content usage among all COVID Coach users within the observation window (between May 1, 2020, and October 31, 2020).
Key content area (specific in-app action) | Unique app users, n (%)a | Key events, n (%)b | Totals per app user, mean (SD); range |
Manage Stress (tried at least one tool) | 28,009 (56.8) | 325,691 (70.4) | 11.63 (30.33); 1-2124 |
Learn (viewed at least one topic) | 10,124 (20.5) | 52,123 (11.3) | 5.15 (8.09); 1-267 |
Mood Check (entered and rated at least one goal or completed at least one assessment ) | 13,510 (27.4) | 47,821 (10.3) | 3.54 (12.52); 1-1008 |
Find Resources (viewed at least one specific subsection) | 9418 (19.1) | 37,016 (8.0) | 3.93 (7.82); 1-329 |
aTotal number of unique app users during the observation window=49,297. Percentage of total app users. Percentages in this column will not sum to 100% because app users could have completed actions across the four types of key content areas.
bTotal key app events during the observation window=462,651. Percentage of total key app events.
Overall, the psychoeducation content within the
In total, core activities within the
Across the eleven subsections within
Detailed key content usage among all COVID Coach users within the observation window (between May 1, 2020, and October 31, 2020).
Key content area | Unique app users, n (%)a | Key app events, n (%)b | |
|
|||
Ambient Sounds (an audio-only tool with no narration) | 7041 (14.3) | 18,493 (4.0) | |
Deep Breathing (an audio-guided exercise) | 7870 (16.0) | 16,011 (3.5) | |
Change Your Perspective (a tool with tips for how to replace negative thoughts with more helpful ones) | 6721 (13.6) | 12,480 (2.7) | |
Muscle Relaxation (an audio-guided exercise focused on relaxing distinct core body parts) | 6037 (12.2) | 11,599 (2.5) | |
Grounding (a tool with tips on how to stay connected to the present moment and surroundings) | 5718 (11.6) | 9767 (2.1) | |
|
|||
Prioritizing Yourself, Right Now | 2131 (4.3) | 2811 (0.6) | |
Managing Irritability | 1856 (3.8) | 2281 (0.5) | |
Finding Humor | 1506 (3.1) | 1817 (0.4) | |
Finding Calm | 1442 (2.9) | 1813 (0.4) | |
Sleep | 1184 (2.4) | 1511 (0.3) | |
|
|||
Finding Local Resources (for locating state-specific COVID-19 guidelines and information) | 2927 (5.9) | 6451 (1.4) | |
Meeting Your Needs (for basic needs support) | 3176 (6.4) | 6223 (1.3) | |
Mobile Apps to Support Mental Health (information about other free apps to support mental health) | 2461 (5.0) | 3748 (0.8) | |
|
|||
Track Mood (PHQ-9c) | 7698 (15.6) | 11,732 (2.5) | |
Track Anxiety (GAD-7d) | 8115 (16.5) | 11,649 (2.5) | |
Track Well-Being (WEMWBSe) | 6151 (12.5) | 8860 (1.9) | |
Track PTSDf Symptoms (PCL-5g) | 3568 (7.2) | 5327 (1.2) |
aTotal number of unique app users during the observation window=49,297. Percentage of total app users.
bTotal key app events during the observation window=462,651. Percentage of total key app events.
cPHQ-9: Patient Health Questionnaire-9.
dGAD-7: Generalized Anxiety Disorder-7.
eWEMWBS: Warwick-Edinburgh Mental Well-Being Scale.
fPTSD: posttraumatic stress disorder.
gPCL-5: Posttraumatic Stress Disorder Checklist-5.
Among the 49,297 app user install codes present in the observation window, 86.8% (n=42,783) of COVID Coach users had their first day of app use occur within the observation window. Thus, for the analyses presented in this section, usage patterns will be restricted to only the app users and events associated with those whose first day of app use occurred during the observation window.
Nearly half of COVID Coach users used the app for a single day (n=20,793, 48.6%), and an additional 32.3% (n=13,818) used the app for 2 or 3 days in total. Less than 2% of the sample (n=709) used the app for 15 or more distinct days (
Total number of distinct days of COVID Coach use, by month of first app use.
Month of first app use | Frequency of users per distinct day, n (%) | |||||
1 day only | 2 days | 3 days | 4-6 days | 7-14 days | ≥15 days | |
May | 6573 (42.67) | 3406 (22.11) | 1891 (12.28) | 2137 (13.87) | 1049 (6.81) | 348 (2.26) |
June | 2533 (44.26) | 1205 (21.10) | 693 (12.14) | 770 (13.49) | 372 (6.51) | 137 (2.40) |
July | 2939 (48.79) | 1246 (20.68) | 640 (10.62) | 737 (12.23) | 362 (6.01) | 100 (1.66) |
August | 3165 (51.31) | 1267 (20.54) | 648 (10.51) | 698 (11.32) | 314 (5.09) | 76 (1.23) |
September | 2783 (53.66) | 1119 (21.58) | 548 (10.57) | 503 (9.70) | 190 (3.66) | 43 (0.83) |
October | 2800 (65.25) | 849 (19.79) | 306 (7.13) | 273 (6.36) | 58 (1.35) | 5 (0.12) |
All | 20,793 (48.6) | 9092 (21.25) | 4726 (11.05) | 5118 (11.96) | 2345 (5.48) | 709 (1.66) |
Analysis of days between first and second app use among COVID Coach users with at least 2 distinct days of app use, by month of first use.
First month of app use | Users that returned at least once, n | First return within 7 days, n (%) | First return within 8-14 days, n (%) | First return within 15-30 days, n (%) | First return within 31-60 days, n (%) | First return within 61-90 days, n (%) | First return after more than 90 days, n (%) |
May | 8831 | 4956 (56.12) | 1071 (12.13) | 1107 (12.54) | 870 (9.85) | 391 (4.43) | 436 (4.94) |
June | 3177 | 1786 (56) | 403 (12.68) | 457 (14.38) | 298 (9.38) | 118 (3.71) | 115 (3.62) |
July | 3085 | 1922 (62.30) | 385 (12.48) | 366 (11.86) | 232 (7.52) | 130 (4.21) | 50 (1.62) |
On both the first and second days of app use (see
Of those who tried at least one
Additionally, usage patterns among individuals who completed an assessment on the first day of app use were significantly different than those who did not complete an assessment on their first day. On average, individuals who completed at least one assessment on their first day of app use utilized COVID Coach for more unique days within the observation window (mean 3.29 days, SD 5.44) compared to individuals who did not complete an assessment on the first day (mean 2.66 days, SD 4.37;
Comparison of key content area usage, by first and second day of app use.
Number of key content areas accessed | App users | ||
First day of app use (n=42,783), n (%) | Second day of app use (n=21,990), n (%) | ||
|
|||
Completed at least one action within all four key content areas | 650 (1.5) | 192 (0.9) | |
|
|||
Manage Stress (with one or two other key areas; tried at least one tool and completed another action within one or two other key areas) | 7953 (18.6) | 3143 (14.3) | |
Two or three key areas (excluding Manage Stress; completed at least one action within two or more of the Learn, Mood Check, or Find Resources sections) | 1129 (2.6) | 405 (1.8) | |
|
|||
Manage Stress only (only tried at least one tool) | 11,419 (26.7) | 5867 (26.7) | |
Mood Check only (only completed at least one goal rating or assessment) | 2677 (6.3) | 1355 (6.2) | |
Find Resources only (only viewed at least one resource subsection) | 1196 (2.8) | 660 (3.0) | |
Learn only (only viewed at least one learn topic) | 805 (1.9) | 485 (2.2) | |
|
|||
Did not complete an action within any of the four key areas | 16,954 (39.6) | 9883 (44.9) |
Baseline well-being among individuals using COVID Coach appeared to be relatively low and decreased over time. Among app users who completed a WEMWBS assessment on their first day of app use (n=3558, 8.32% of all users whose first day of app use occurred during the observation window), average well-being scores, by month, were all less than 42, which has been used as a cut-off to identify low well-being [
Symptoms of anxiety, depression, and PTSD were prevalent among app users. For all app users who completed a GAD-7 assessment on their first day of app use (n=4870; 11.4% of users), 12.8% (n=625) had scores suggesting minimal anxiety (total score=0-4), 32.1% (n=1565) endorsed mild levels of anxiety (total score=5-9), 25.2% (n=1225) indicated moderate levels of anxiety (total score=10-14), and 29.9% (n=1455) of scores fell into the severe symptom range (total score=15 or higher).
Among app users who completed a PHQ-9 on their first day of app use (n=4548, 10.6% of users), 16.5% (n=749) had scores suggesting minimal depression (total score=0-4), 28.9% (n=1312) endorsed mild levels of depression (total score=5-9), 25.0% (n=1136) indicated moderate levels of depression (total score=10-14), 17.5% (n=795) endorsed moderately severe levels of depression (total score=15-19), and 12.2% (n=556) of scores fell into the severe symptom range (total score=20 or higher).
Unlike the GAD-7 and the PHQ-9, the PCL-5 does not have symptom severity categorizations. However, among app users who completed a PCL-5 on their first day of app use (n=2064, 4.8% of users), the majority of individuals who completed the assessment (n=1234, 59.8%) had a total score ≥33, which is consistent with significant PTSD symptoms.
Baseline PTSD symptoms predicted returning to the app for a second day. Among individuals with a baseline PCL-5 score of 33 or greater, 62.5% returned to the app for a second day of use, compared to only 56.4% of individuals with scores below 33 (
We conducted regression analyses to examine the relationship between baseline mental health symptoms and unique days of app usage. Depression and PTSD symptoms were predictive of the total number of unique days of app use. With respect to depression symptoms, we utilized the group with minimal symptoms as the reference group in comparison to those with mild, moderate, moderately severe, and severe symptoms. On average, those with moderate levels of depression on their first day of app use returned to the app for a greater number of days (mean 3.72 days) than those with minimal symptoms of depression (mean 3.08 days;
This exploration of COVID Coach usage among the general population suggests that mobile apps may have the reach and accessibility necessary to be a useful medium for disseminating mental health information and resources to individuals experiencing stress related to the COVID-19 pandemic.
Between May 1, 2020, and October 31, 2020, the app was used by nearly 50,000 individuals, and daily active usage has remained steady over time. In addition to the total number of individuals reached, the key content within the app was utilized in over 450,000 instances. The stress management tools were most frequently used with over 28,000 users utilizing individual tools over 300,000 times. Further, each of the other three key content areas in the app were accessed tens of thousands of times by tens of thousands of users. This reach and scalability of COVID Coach across the general population is an example of how digital mental health tools can become successfully integrated into disaster response strategies. From a public mental health perspective (eg, [
Importantly, COVID Coach appears to be reaching individuals in need of mental health resources. On average, among app users who completed assessments during their first day of use, well-being was low, and the majority of individuals were indicating greater than minimal symptoms of anxiety, depression, and PTSD. Additionally, among app users who identified challenges they are facing, the majority reported difficulties with managing stress, troubles with sleep, and feelings of loneliness. We cannot determine if individuals utilizing COVID Coach are representative of the general population, but elevated levels of anxiety, depression, and posttraumatic stress are consistent with other research conducted during the pandemic [
Although overall app utilization data suggested considerable reach, engagement proved to be less consistent. Our analyses revealed that the majority of COVID Coach users (80.9%) utilized the app on ≤3 days. This finding is consistent with research indicating that self-management apps for mental health are often not used over extended periods of time [
This research also provides some guidance on how engagement might be encouraged in future app versions. In general, app users that completed actions within the key content areas on the first day of app use were more likely to return for a second day of app use. More specifically, users that completed an assessment on the first day of app use were significantly more likely to use the app for a greater number of days and to use a greater number of stress management tools than app users who did not complete an assessment on the first day of app use. These findings suggest that finding ways to motivate users to complete actions within key areas on their first day of app use, particularly tools and assessments, may be one way to enhance engagement and retention. For example, having recommendations for a tool or assessment to try, easily accessible from the app home screen, may encourage users to try a specific in-app activity. Additionally, the onboarding sequence could include a few brief questions to help tailor in-app recommendations to the user’s intentions and preferences, and guide them through the process of setting customized goals for using the features within the app most relevant to them. Lastly, finding ways to regularly disseminate and highlight new app content (eg, managing stress around prolonged distance learning, vaccine information) may encourage users to return to the app more frequently.
Because COVID Coach does not collect any identifying information, we cannot say anything about the populations that we have reached, other than what we can characterize based upon in-app actions. Future research that permits collection of identifying information is needed, particularly given the disproportionate impact the pandemic has had on vulnerable groups of people. A Spanish version of COVID Coach has recently been released, and plans for data collection on app usage within Spanish-speaking populations are underway.
Additionally, we utilize the unique install codes as a proxy for an individual user. We assume that most individuals do not delete and reinstall the app multiple times. However, if an individual were to download COVID Coach on more than one mobile device, or delete it and reinstall, each of those installations would be assigned a unique install code, and would appear as a new user.
Although the app includes assessments for individuals to self-monitor well-being and symptoms of anxiety, depression, and PTSD, it is difficult to reliably measure change in these constructs via the app, due to the naturalistic nature of this study and the changing landscape of the pandemic over time. It is important to highlight that even though a score of 33 or higher on the PCL-5 is suggestive of PTSD, the assessment questions in the app do not ask app users to respond to the questions while focusing on a particular traumatic incident, so caution in interpreting the meaning of these scores is warranted. Because the PCL-5 refers to “the stressful experience” in each item, in the context of COVID Coach, the PCL-5 may be capturing overall levels of distress. While desirable, we also did not have a way to measure other potential proxy variables of interest such as coping self-efficacy, perceived helpfulness of the app, improved opinions about mental health care, or reduction in stress related to enhanced support access, as these cannot be determined solely by in-app usage data.
Future research is needed to better understand who is interested in public mental health apps like COVID Coach, what their primary goals are for using the app, which outcomes are most useful in understanding engagement patterns, and how successful usage is defined. For example, someone may use the app only once, find the exact resource they need, and not use the app again, whereas someone else may be experiencing significant stress, use tools in moments of distress, and track mental health symptoms on a weekly basis. Findings from this type of research could be used to advance the science of mobile mental health and also be directly applied to a suite of publicly available apps that have been downloaded over 4 million times and are in widespread use across the VA, the largest health care organization in the United States.
As the mental health impacts of the pandemic continue to be widespread and increasing, digital health resources, such as apps like COVID Coach, are a scalable way to provide evidence-informed tools and resources. We believe that this is the first evaluation of a mobile mental health app designed specifically for use during the COVID-19 pandemic. This work shows that tens of thousands of people are accessing the app, with a particular focus on the tools for stress and coping. Such rapid uptake of a public mobile mental health app is unprecedented and signals perceived value. Specially, the findings from this evaluation suggest that apps may play a helpful role in providing mental health resources in the context of a public health disaster.
Future research should attempt to elucidate for whom and under what conditions the app is most helpful, and how to increase and sustain engagement. Additional areas of focus should include how to optimize the app for populations impacted by disparities related to mental health literacy, digital literacy, and stigma around mental health care. As noted by many mHealth (mobile health) scholars [
COVID Coach screenshots.
analysis of variance
Black, Indigenous, people of color
Generalized Anxiety Disorder-7
JavaScript Object Notation
mobile health
Posttraumatic Stress Disorder Checklist-5
Patient Health Questionnaire-9
posttraumatic stress disorder
Coordinated Universal Time
Veterans Affairs
Warwick-Edinburgh Mental Well-Being Scale
This paper was not sponsored by any funder. The views expressed in this work are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.
All authors contributed to the conceptualization, writing, and editing of the paper.
None declared.