The World Health Organization has recognized the importance of assessing population-level mental health during the COVID-19 pandemic. More than 463 articles on the mental health impact of COVID-19 have been published in 2020, and several more mental health surveys are underway [1]. Most of these studies evaluated the psychiatric symptoms among the general public and health care workers, and some others evaluated patients with COVID-19 or those with psychiatric disorders [2]. In a global crisis such as the COVID-19 pandemic, time-sensitive policy decision-making underscores the importance of fostering an agile empirical approach that can monitor population-level mental health in a timely manner. Any change in the mental health impact of COVID-19 is likely to be dynamic. For example, the 20% decrease in suicide rate in Japan observed during the early stage of the pandemic seemed to have reversed in August 2020, when a 7.7% rise was reported [3]. Therefore, a timely surveillance method is urgently needed to track the impact of COVID-19 on public mental health. This brief systematic review focused on studies on the general public conducted during the COVID-19 pandemic and analyzed the efficiency and quality of data collection in the existing literature.

We used the following search strings to select relevant articles in the PubMed database: ((COVID-19) OR (SARS-CoV-2)) AND ((Mental health) OR (psychological) OR (psychiatry)). We screened the titles, abstracts, and texts of the published articles to exclude irrelevant studies. We also excluded articles published on preprint platforms such as bioRxiv, because we were unable to analyze the time between the receipt of the articles and their availability online. A PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram detailing the study retrieval process is shown in Figure 1.

We used the Newcastle-Ottawa Scale to evaluate the quality of each research article. The Newcastle-Ottawa Scale includes four criteria for selection bias, with scores ranging between 0 and 5; scores <3 points are regarded as high risk for selection bias in a cross-sectional study [1,4]. The four quality assessment criteria were as follows: sample representativeness (1 point), predetermined and satisfactory sample size (1 point), comparability between respondents and nonrespondents (1 point), and ascertainment of the measurement tool (2 points for the validated measurement tool; 1 point for the nonvalidated measurement tool, but the tool is available or described).

In all, we found 37 relevant mental health surveys of the general public during the COVID-19 pandemic by searching the PubMed database, as of July 10, 2020. All these public mental health survey studies examined had a cross-sectional study design. The journals efficiently made these articles available online in an average of 18.7 (range 1-64) days from the date they were received. The average duration of recruitment period of these studies was 9.2 (range 2-35) days, and the average sample size was 5137 (range 100-56,679). The correlation coefficient (r=0.164, P=.40) between these durations and the corresponding sample sizes suggested that the samples in the studies selected from the existing literature were collected in an efficient manner; even larger sample sizes did not require proportionately longer to collect the data. Although there was a potential publication bias in the recruitment or data collection period, it is noteworthy that approximately 92% (34/37) of the studies reported the use of a web-based or an app-based survey, thereby promoting the efficiency of large-scale data collection, especially during the COVID-19 pandemic.

Unlike most studies on health care workers, which have a low risk of selection bias [1], our review showed that 73% (27/37) of the selected studies evaluating the mental health impact of COVID-19 on the general public had Newcastle-Ottawa Scale scores of <3 points, which suggests that these studies were of very low quality to be considered for inclusion in a meta-analysis. All of the selected studies met the criterion of validated measurement tools, but only 8 studies met the criterion of sample representativeness, 5 met the criterion of justified sample size, and 1 met the criterion of comparability between respondents and nonrespondents (Table 1).

The general public surveys examined in this review most commonly used the web-based format to promote efficiency of large-scale data collection, especially in the countries with significant COVID-19 outbreaks that had introduced social distancing or lockdown measures. However, the web-based survey approach has an inherent methodological limitation to show the comparability between respondents and nonrespondents. The sample representativeness of the target population is critical for selection bias. Web-based survey platforms using sampling methods could avoid such selection bias. Two published surveys [6,11] used Qualtrics Panels, a web-based survey platform that uses quota sampling methods to identify participants who meet a given study’s eligibility criteria, with a target to recruit a sample that is demographically similar to the 2010 United States census distributions for age, sex, and race or ethnicity (with a ±10% variation). Qualtrics Panels maintains a database of several million US residents who have volunteered to participate in periodic survey-based research. Another study [20] used AmeriSpeak, a probability-based research panel designed to be representative of the adult US population. AmeriSpeak data is sourced from National Opinion Research Center’s area probability sample and a US postal service address–based sample covering approximately 97% of all US households. In this national survey study [20], a higher proportion (13.6%) of the US adult sample reported symptoms of serious psychological distress in 2020 than in 2018 (3.9%).

The limitations of cross-sectional studies should also be noted when interpreting the role of the COVID-19 outbreak on the mental health of the general public. These epidemiological studies identified general risk factors during the COVID-19 outbreak, rather than the specific impacts of COVID-19 on the mental health of the public. For example, a cross-sectional study of 4827 participants [12] showed that lower education levels and higher social media exposure result in a high risk of individuals developing depressive symptoms, but these findings did not suggest that COVID-19 had an impact on the public’s mental health. There was only one longitudinal study [35], which conducted two-wave surveys during the initial COVID-19 outbreak (ie, January 31 to February 2, 2020) and during the peak 4 weeks later (ie, February 28 to March 1). Although this study recruited 1210 and 861 participants within 3 days by using a web-based survey approach, no significant longitudinal changes in stress, anxiety, or depression levels were observed. These findings might result from a lack of baseline data, few follow-ups with the same respondents (333/1210, 27.52%), or insufficient temporal resolution to detect mental health changes. Only one study compared the prevalence of symptoms of serious psychological distress before and during the COVID-19 pandemic by using an identical measure. This survey, which used AmeriSpeak for sample recruitment, showed that the prevalence of serious psychological distress symptoms among US adults was higher during the COVID-19 pandemic in 2020 than that reported in the 2018 National Health Interview Survey [20]. However, this study did not monitor the changes in population-level mental health through the unfolding pandemic. Timely mental health survey studies with a high temporal resolution, such as internet search data (eg, Google Trends) [42] and sentiment analysis on social media (eg, Weibo posts) [43], are warranted in the future to monitor the long-term impacts of the fast-moving COVID-19 outbreak.

In conclusion, this systematic review found that the data collection methods used in the public mental health surveys in the existing literature were efficient but generally had a high risk of bias. Therefore, following recommendations to avoid selection bias, or to apply novel methodologies considering both longitudinal design and high temporal resolution, would help provide a strong basis for the formation of national mental health policies.