Background: In the face of the COVID-19 pandemic, the German government and the 16 German federal states implemented a variety of nonpharmaceutical interventions (NPIs) to decelerate the spread of the SARS-CoV-2 virus and thus prevent a collapse of the health care system. These measures comprised, among others, social distancing, the temporary closure of shops and schools, and a ban of large public gatherings and meetings with people not living in the same household.
Objective: It is fair to assume that the issued NPIs have heavily affected social life and psychological functioning. We therefore aimed to examine possible effects of this lockdown in conjunction with daily new infections and the state of the national economy on people’s interests, motives, and other psychological states.
Methods: We derived 249 keywords from the Google Trends database, tapping into 27 empirically and rationally selected psychological domains. To overcome issues with reliability and specificity of individual indicator variables, broad factors were derived by means of time series factor analysis. All domains were subjected to a change point analysis and time series regression analysis with infection rates, NPIs, and the state of the economy as predictors. All keywords and analyses were preregistered prior to analysis.
Results: With the pandemic arriving in Germany, significant increases in people’s search interests were observed in virtually all domains. Although most of the changes were short-lasting, each had a distinguishable onset during the lockdown period. Regression analysis of the Google Trends data confirmed pronounced autoregressive effects for the investigated variables, while forecasting by means of the tested predictors (ie, daily new infections, NPIs, and the state of economy) was moderate at best.
Conclusions: Our findings indicate that people’s interests, motives, and psychological states are heavily affected in times of crisis and lockdown. Specifically, disease- and virus-related domains (eg, pandemic disease, symptoms) peaked early, whereas personal health strategies (eg, masks, homeschooling) peaked later during the lockdown. Domains addressing social life and psychosocial functioning showed long-term increases in public interest. Renovation was the only domain to show a decrease in search interest with the onset of the lockdown. As changes in search behavior are consistent over multiple domains, a Google Trends analysis may provide information for policy makers on how to adapt and develop intervention, information, and prevention strategies, especially when NPIs are in effect.
On March 11, 2020, the World Health Organization (WHO) declared the COVID-19 outbreak a pandemic, as almost all countries of the world were affected . One month earlier, the WHO announced the official names of this coronavirus and the disease it causes as SARS-CoV-2 and COVID-19, respectively [ ]. As of October 3, 2020, there have been 296,958 reported cases in Germany [ ] and 34,495,176 reported cases worldwide [ ].
With increasing case numbers and death counts comes the necessity to monitor and slow the further spread of the pandemic . There is strong evidence that human-to-human transmission is the reason the initial outbreak in Wuhan, China, became global; thus, it is imperative to prevent secondary infections [ ] (eg, by means of restricting air traffic [ ] and other nonpharmaceutical interventions [NPIs]) in the absence of a vaccine [ ]. NPIs are used to pursue two strategies—mitigation and suppression of further spread—which can be achieved by a combination of multiple population-wide measures, such as social distancing by the entire population; home isolation and quarantine of cases and their household members; banning mass gatherings, and closing schools and universities [ , ]. The implementation of these measures seems to slow the spread of the pandemic, thus preventing thousands of deaths [ , ]. As data for monitoring disease spread is difficult to obtain in the early stages of a pandemic, the use of internet data has become increasingly valuable, with the emergence of two research disciplines: infodemiology and infoveillance [ ]. Although the former is a research discipline and methodology studying the determinants and distribution of health information [ ], the latter is defined as the longitudinal tracking of infodemiology metrics with the goal of surveillance and trend analysis [ ]. Both can help policy makers in developing prevention and information strategies. As information influences knowledge, behavior changes, and health outcomes [ , ], the internet can be used as a handy tool to monitor public search behavior [ - ] or public awareness [ ] of health care issues, allowing policy makers to quantify knowledge translation gaps [ ], identify misleading information circulation [ , ] or possible emerging local outbreaks [ ], assess people’s needs [ ], derive suitable interventions [ - ], and develop enhanced communication strategies especially for at-risk subpopulations [ ].
To inform policy makers about crisis-related developments, researchers have examined changes in search behavior during the COVID-19 pandemic [, ]. Google Trends data was used to monitor public interest in the pandemic from January to March 2020 by conceptualizing the demand for information as a single search term (“coronavirus”), showing worldwide peaks in people’s interest at the end of January as well as shortly after COVID-19 was declared a pandemic. Pearson correlations indicated a moderate to high relationship between different countries regarding search behavior [ ], although some studies (eg, Hu et al [ ]) suggest differences during the first peak between countries and even various subregions of countries. Typically, announcements of the first local COVID-19 case led to a widespread increase in search interest as people sought information about the COVID-19 pandemic [ ]. There is evidence, though, that COVID-19 terms trending in Italy elicited differential peaks, showing that regions most heavily affected in terms of infections and deaths were not always the first to display changes in search interest [ ], as local and international events are capable of influencing public interest [ ]. However, the number of recent COVID-19 infections and deaths is associated with changes in public interest, which becomes apparent by maximum correlations when a lead of approximately –11.5 days is applied [ ]. This lagged nature of relationships has also been found for other COVID-19–related search tags (eg, the frequency of search queries for loss of smell [ , ] and taste [ , ] as key symptoms of COVID-19, or public interest in “pneumonia” [ ] or “high temperature”, “cough”, and “diabetes” [ ], as well as “insomnia”, a key signal for mental distress [ ]). The positive relationship seems to be consistent across different social media platforms and search engines, such as Baidu and Sina Weibo Indices [ ], as well as with other diseases (eg, the flu outbreak in 2004 [ ]). However, Mavragani [ ], who used Google Trends data with the search tag “coronavirus” to study European countries (ie, Italy, Spain, France, Germany, and the United Kingdom) found moderate to high positive Pearson correlations between search queries and the number of total and daily new cases for all countries except Italy. An in-depth analysis of multiple time frames indicated a declining trend of the Pearson correlation coefficient over time, highlighting an early interest phenomenon. Overall, the increase of attention is typically short-lived [ , ], even when accompanied by NPIs issued by policy makers [ ].
Given that Google Trends data can indicate population health literacy, Google Trends queries for “wash hands” and “face mask” were found to correlate with a lower spreading rate of COVID-19 cases in 21 countries: the logarithmic increase of case numbers correlated negatively with the number of days with high search volume . Google Trends data could moderately predict new cases, although predictions were not found to be very precise [ ]. Nonetheless, COVID-19 symptoms showed some predictive value, especially when virus diffusion was monitored by predicting the number of deaths [ ]. Even real-time forecasting of the COVID-19 outbreak has been shown to be possible [ , ].
However, changes in search behavior are not only limited to disease symptoms (eg, “anosmia” ). Previous research has similarly shown that COVID-19 has affected the social, economic, and psychological well-being of humanity, pervading broad domains of social and daily life, such as COVID-19 awareness [ , ], psychological stress (eg, “anxiety”, “depression”, “therapy” [ ]), food supply (eg, “sourdough”, “restaurants”, “baking” [ ]), economic stressors (eg, “mortgage”, “homeloan”), social stressors (eg, “divorce”, “liquor”), and treatment seeking (eg, “cognitive therapy”, “counselling” [ ]), as well as quarantine and even conspiracy theories [ ]. An analysis of Twitter data identified four main domains of interest for people: the origin of the virus, its sources, its impact on people, countries, and economies, as well as ways of mitigating the risk of infection. The latter suggests that Twitter users are keen to learn and also share their knowledge with friends and followers, while also discussing the negative consequences of the COVID-19 pandemic and its emotional and psychological impact [ ]. Further research found changes in stress-related and food-related searches, indicating that the outbreak may have had a significant impact on both people’s stress levels and daily routines [ ].
Despite a growing body of research studying search behavior regarding COVID-19 and its symptoms in conjunction with infection and death counts, there seem to be limited research activities targeting the effects of the issued NPIs. One early study used Google Trends data from January to March 2020 to examine the impact of the pandemic and the issued NPIs on people’s concerns, indexed by changes in search frequency about mental health issues as well as financial and work-related consequences. They found an increase in financial and work-related queries, indicating that people are well aware of further consequences, such as job loss. There is also some evidence that queries for “depression” and “suicide” decreased immediately after the announced pandemic, but increased shortly after the first wave . Another study stressed the potential damage to a population’s well-being as a consequence of the lockdown. It used Google Trends data to test for changes in people’s well-being and found a substantial increase in search intensity for boredom in European countries and the United States, which was accompanied by a higher search volume for loneliness, worry, and sadness, while searches for stress, suicide, and divorce declined [ ].
Since previous research has shown that the COVID-19 pandemic has resulted in changes in public interest [, , ] beyond the expected search behavior regarding COVID-19–related symptoms [ ], we conducted this study to investigate the effects of the pandemic and the corresponding lockdown using a broad set of psychological domains and interests. As Google Trends data can function as a proxy for psychological states and behaviors [ ], search interests may indicate changes in social and daily life. Our 27 domains of interest range from COVID-19 restrictions to online shopping, sexual interests, and psychosocial impact. The German government and the 16 German federal states issued a cluster of NPIs to decelerate the spread of COVID-19. These measures comprised, among others, social distancing, the temporary closure of shops and schools, and banning large public gatherings and meetings with people not living in the same household [ ]. As prior studies suggested, it seems fair to assume that all measures heavily affected social life and psychological functioning [ , - ]. Thus, we expected significant changes in search behavior during the emerging pandemic and the subsequently issued lockdown period for all observed domains. Further, daily new infections were tested as a predictor, given that infection and death counts have been shown to be associated with an increase in public interest [ , ]. Additionally, the state of the national economy was investigated as a predictor, as it may affect people’s employment situation and related psychological anticipations, worries, and the need to search for COVID-19–related information. In turn, changes in search queries and their relevant predictors can be used to inform policy makers, allowing them to tailor information and health prevention strategies [ - ].
Psychological Domains and Indicators of Interest
As preregistered , we planned to examine 27 psychologically meaningful domains that may possibly have been affected by the COVID-19 pandemic, the NPIs, or associated economic developments. For each domain, at least three search terms were selected as indicators. For instance, the domain “pandemic disease” comprised “coronavirus”, “covid sars”, or “covid-19” as Google search tags. Descriptions of all domains are given in . For all search terms, refer to .
|Pandemic disease||People’s interest in the virus and the disease it causes|
|Health care institutions||German and international institutions as well as experts providing information on the pandemic|
|Political leaders||Names of leading German politicians whose responsibilities comprise dealing with the pandemic|
|Infection||Information concerning the rapidity and method of virus spread|
|Symptoms||Possible symptoms experienced by patients with COVID-19 indicating a virus infection|
|Information seeking||Sources and methods people could use to obtain an overview of the development of the pandemic|
|Testing||Questions pertaining to the capacities of testing, including locations and general opportunities to have oneself tested|
|Hand hygiene||Possibilities of keeping one’s hands virus-free|
|Mask||Interest in the different types of masks, their effectiveness, and the current regulations on wearing a mask|
|Disinfectant||Possible substances used for disinfection and recommendations for appropriate disinfection|
|Convenience goods||Groceries or household goods that were nearly sold out due to panic buying and hoarding|
|Vaccination||Interest in vaccination possibilities and fear of a compulsory vaccination|
|Parenting and childcare||Challenges parents have to deal with during the pandemic|
|COVID-19 restrictions||Information about the current restrictions and rules of conduct|
|COVID-19 relaxations||Information about current and future possibilities and relaxations of the issued restrictions|
|Economic impact||Economic shortcomings and development as consequences of the COVID-19 restrictions|
|Sexual interest||Changes in sexual interest due to the COVID-19 crisis|
|Social life||Possibilities for people to socialize while following COVID-19 restrictions such as social distancing|
|Homeschooling||Information about new ways of educating students at home while schools were closed|
|Business communication||Different ways for employees to stay in contact with their colleagues and to be able to work from home|
|Hobbies and sports||Activities that became popular during the lockdown as they could be done without breaking restrictions|
|Renovation||Renovation activities that could be done at home during the pandemic|
|Online shopping||Names of common online sellers|
|Dispatching||Terms and information about the shipment of items|
|Psychosocial impact||Negative consequences that arose due to the COVID-19 restrictions|
|Conspiracy theories||Several conspiracy theories related to the source and spread of the pandemic|
|Government support||Terms and information about financial aid due to closures and severe restrictions|
Retrieving Google Trends Data
Each domain was entered in the Google Trends database to reveal the number of Google search queries for a given time period and geolocation using the gtrends package  for the statistical language R [ ]. Before downloading, we thoroughly studied the framework for using Google Trends data, which highlights the choice of region, time period, and geolocation [ ]. As Google allows one to set different geolocations at the country and federal state level within a specified period, we retrieved Google Trends data separately for each search tag for 17 geolocations (ie, Germany and its 16 federal states) for the period from January 6 to September 1, 2020. We chose this period to allow for a sufficient period before the first cases occurred in Germany as a baseline estimate. By default, Google standardizes the queries per day by the maximum of queries within the given period, multiplying all values by 100. High interest in a query is expressed by 100, whereas minor interest or missing data on queries is expressed by 0. Values below 1 were recoded as 1.
We retrieved Google Trends data for all domains for all 17 geolocations except for 32 search tags falling into 16 domains. An overview of the domains and search tags for which we could retrieve Google Trends data is provided in. For the analysis, we used only keywords available for all federal states. As there were differences between the retrieved time series for the geolocation “DE” for Germany and the aggregated time series across all federal states, we decided to use the aggregated time series for further analysis.
Retrieving Daily New Infections Data
As the Robert Koch Institute (RKI), a German federal government agency and research institute responsible for disease control and prevention, issued daily situation reports concerning COVID-19  even before the official declaration of the pandemic, we made use of the online dashboard application programming interface (API) to retrieve daily data, comprising infection and death counts for each federal state as well as Germany as a whole from January 6 to September 1, 2020. Maps of the German federal states together with 7-day incidence rates are provided in , along with summary statistics for, for example, population density, cumulative infections, and incidence rates in .
Retrieving Economic Data
The truck toll mileage index (TTMI) provides an early and sensitive day-to-day proxy for the state of industrial production. It is computed by using process data of the toll system for trucks in Germany . According to the German Federal Statistical Office (Statistisches Bundesamt), the TTMI can serve as a highly accurate indicator of the economic state with a lag of 5 to 9 days. We retrieved the data for the period from January 6 to September 1, 2020.
Retrieving NPI Data
In the face of the COVID-19 pandemic, the German government and its federal states issued 14 NPIs to curb the spread of COVID-19. The measures included limitations regarding leaving home without reason (ie, only for grocery shopping or a doctor’s appointment), recommendations to keep social distance (>1.5m), the obligation to wear a mask in stores and when using public transport, closure of nonessential shops, closure of hairdressers and close contact services (eg, massage studios), closure of nonfood shops (eg, bicycle and hardware stores), closure of zoos, the prohibition of public demonstrations; closure of schools, playgrounds, kindergartens, and day care facilities; closure of religious institutions, and prohibition of meeting in public with persons not belonging to the same household. The Leibniz Institute for Psychology (ZPID) provided coded daily data for all relevant NPIs with their onset, separated by federal state, from March 8 to June 26, 2020 . We will subsequently refer to this period as the lockdown period.
Data Reduction and Preprocessing
Within each domain, a time series factor analysis (TSFA)  was used for Germany to test if one hypothesized dominant factor accounts for the covariation of indicators in the respective domain. First, we extracted the moving average for 7 days (the so-called trend) for all search terms for a domain. Second, we z-standardized the time series to attain loadings, which can be interpreted as correlations (–1≤λ≤1), and estimated the factor analysis with a single assumed factor. We used only the trend information of time series as it represents the long-term process and behavior of the time series [ ]. Indicator variables were selected when they had positive loadings on the first factor, thereby iteratively excluding indicators with negative loadings. We excluded 20 search terms falling into 10 domains ( ).
For all domains, the scree plots resulting from the selection process are presented in. Next, factor scores were calculated using the loadings on the first factor as weights to aggregate a single time series for each keyword per federal state, retrieving a mean time series for each domain. We then deseasonalized the time series for each domain, supposing an additive model with weekly seasonality for every time series.
Since the NPIs were issued at different times, we used TSFA to identify factors that describe their covariation. We differenced every time series, so they corresponded with daily change scores, prior to computing the TSFA. Scree plots could be interpreted as indicating a two- or three-factorial structure. The two-factor model could be more consistently interpreted, yielding one factor of “regulation of outdoor activities” (NPI Outdoor) and another factor of “regulation of social life” (NPI Social). Corresponding with the selection criteria applied to Google Trends data, we excluded indicator time series with negative loadings. We calculated factor scores using the loadings as weights to aggregate single time series for each NPI, resulting in a single mean time series for each NPI factor. Details concerning indicators and their respective loadings are given in.
|NPI||Regulation of outdoor activities (NPI Outdoor)||Regulation of social life (NPI Social)|
|Prohibition to leave home||N/Ab||.91|
|Rule to wear a mask in public||.06||N/A|
|Closure of nonessential shops||.28||N/A|
|Closure of hairdressers and close contact services||.02||.58|
|Closure of nonfood shops||.39||.22|
|Closure of zoos||.99||N/A|
|Prohibition of public demonstrations||.32||N/A|
|Closure of schools||.41||.05|
|Closure of religious institutions||.67||.05|
|Prohibition to meet with nonhousehold members||N/A||.73|
|Prohibition to meet with more friends||.14||.69|
|Closure of shops||.93||N/A|
|Closure of kindergartens and day care||.48||.10|
aIndicators with negative loadings were omitted from the factor scores. NPI: nonpharmaceutical intervention. TSFA: time series factor analysis.
bN/A: not applicable.
Visual Inspection and Change Point Analysis
Each aggregated time series per domain and federal state was carefully screened in conjunction with the time series for daily new infections, as well as the economic state and the two NPI factors “regulation of outdoor activities” and “regulation of social life.” As we assumed structural breaks, indicating a major change in the underlying processes of the time series, we proceeded with a change point analysis with an intercept-only model to identify changes in the level for the lockdown period .
Time Series Regression Analysis
For regression analysis, we conceptualized search behavior over time for each federal state as a criterion variable in linear mixed models, testing intercept-only models to identify a possible hierarchical structure as it would be indicated by a substantial intraclass correlation (ICC). Since there were no meaningful ICCs (all ICC<0.05, as preregistered), we proceeded with models for Germany only. We used daily new infections, the NPI factors NPI Outdoor and NPI Social, as well as the TTMI as predictors. Criterion and predictor variables were z-standardized before linear modelling. To account for the autocorrelation in time series, we used an autoregressive integrated moving average (ARIMA) approach to fit the models [, ]. ARIMA models are a form of generalized least squares models with three parameters, namely p, d, and q. The p parameter denotes the order of the autoregressive part AR(p). In an autoregressive model, the values in a time series Yt are predicted by its lagged values Yt-p. For a lag of 1, the AR(1)-model is equal to equation 1.
Yt = β0 + β1Yt-1 + εt (1)
For the moving average part, the parameter q denotes the order of previous error values εt-q used to predict the time series Yt. An MA(1)-model is equal to equation 2.
Yt = β0 + β1 εt-1 + εt (2)
The d parameter stands for “integrated” and denotes the order of differencing that was applied to the time series before an autoregressive moving-average (ARMA) analysis. Differencing is used to adjust for seasonality in the time series and can be repeated multiple times. In summary, an ARIMA model with the parameters p=1, d=1, and q=1 with a predictor X1 and Dt = Yt – Yt-1 is equal to equation 3.
Dt = β0 + β1X1 + β2 εt-1 + β3Dt-1 + εt (3)
In time series regression, predictors are frequently allowed to exert lagged effects. To estimate this lag period, the time parameter is adjusted (shifting the time series) to maximize predictive power. As we were only interested in predicting Google Trends data, we focused on shifting predictors forward from 0 to 21 days. In this case of shifting a time series forward, the value of a time series (Yt) is replaced by its successive value (Yt+1), which is called a lead. We refrained from using the Bayesian information criterion  for model selection as preregistered [ ], since iteratively shifting time series resulted in different combinations of ARIMA parameters, rendering a meaningful interpretation of the respective fit values impossible. Therefore, we selected the lag period for which the predictor exerted a maximal correlation with the criterion variable. The cross-correlation functions for all relevant predictors (ie, daily new infections, NPI factors NPI Outdoor and NPI Social, TTMI) are provided in .
All statistical analyses were performed using the statistical language R  with the package strucchange [ ] for identifying structural breaks and the package lme4 [ ] for linear modeling.
Cross-Correlation Between Domains
Search interests for all 27 domains were moderately to highly correlated (, lower triangle). Exceptions can be noticed for the domains parenting and childcare, COVID-19 relaxations, sexual interest, renovation, and government support. To identify possible lagged relations between domains, we shifted one time series in the range from 0 to 21 days. The maximum correlations obtained within this time-lagged range were highly comparable to the concurrent correlation. Hence, the overall pattern of correlations across domains was hardly affected ( , upper triangle).
Visual and Change Point Analysis
All domains showed pronounced changes in people’s search behavior in the corresponding period from January 6 to September 1, 2020, although the changes were mostly short-lived (and ). Many domains showed different onsets during the COVID-19 crisis. Disease- and virus-related domains (ie, pandemic disease, health care institutions, infection, symptoms, testing, hand hygiene, disinfectant, parenting and childcare, COVID-19 restrictions, and economic impact) show an early onset, but searches decreased quickly. Public interest in political leaders peaked multiple times, while the domains vaccination, social life, homeschooling, hobbies and sports, psychosocial impact, and dispatching seem to be of higher, long-term interest. Interest in other domains seemed to be related to the onset of the NPIs (eg, the domain mask peaked when the obligation to wear masks in shops and public transport was announced). Interest in government support (ie, financial support programs provided by the national government) increased in later stages of the lockdown, while COVID-19 relaxations and conspiracy theories attracted attention primarily in the middle of the lockdown phase. For each domain, noticeable changes in the level (ie, mean for a chosen period of time) indicate significant changes in public interest during the examined lockdown period from March 8 to June 26, 2020, in Germany.
Time Series Regression Analysis
We conceptualized search interests in all 27 domains as criterion variables in ARIMA regression analyses and entered four predictors—namely, daily new infection cases, both NPI factors (NPI Outdoor and NPI Social), and the TTMI as a proxy for the economic state. All analyses were conducted for Germany as a whole, as low ICCs indexed no meaningful heterogeneity between federal states.summarizes the results for the regression part. Overall, all predictors showed a different pattern and, thus, predictive value for different domains. Although new infections seemed to have a delayed effect on public interest (see the “Lead” column), the NPI factors and TTMI had their highest effect directly at lag 0. Almost all models benefitted from the ARIMA approach, as shown in . Most domains needed differencing to account for nonstationarity. Additionally, to handle the correlated residual structure, almost all domains were estimated with an autoregressive or moving average component.
|New infections||NPI Outdoor||NPI Social||TTMI|
|Lead||β1||P value||Lead||β2||P value||Lead||β3||P value||Lead||β4||P value|
|Health care institutions||–9||.23||.02||0||.25||.07||–13||.60||<.001||–13||–.09||.03|
|Parenting and childcare||0||–.32||.02||0||–.25||.18||0||–.17||.42||0||.07||.38|
|Hobbies and sports||0||–.16||.14||–10||–.04||.93||–3||1.07||<.001||–3||–.05||.27|
aLead corresponds to the forward-shifted time series for the corresponding number of days. Zero values indicate the original time series without shift. ARIMA: autoregressive integrated moving average; NPI: nonpharmaceutical intervention; TTMI: truck toll mileage index.
|Domain||ARIMA (p,d,q)||ARIMA term|
|Autoregressive component||Moving average component|
|p||d||q||AR(1)||P value||AR(2)||P value||MA(1)||P value||MA(2)||P value|
|Health care institutions||0||1||1||N/A||N/A||–.64||<.001||N/A|
|Parenting and childcare||0||0||1||N/A||N/A||.40||<.001||N/A|
|Hobbies and sports||1||1||1||.28||.03||N/A||–.85||<.001||N/A|
aThe ARIMA model parameters p, d, and q represent the order of the autoregressive components, the order of differencing of the original time series, and the order of the moving average components, respectively. ARIMA: autoregressive integrated moving average.
bN/A: not applicable.
This study was conducted to test possible effects of the COVID-19 pandemic, in conjunction with lockdown NPIs, daily new infections, and the state of the national economy, on people’s social and psychological interests, going beyond previous research examining only a few domains for a limited timespan [, - ]. Specifically, we tested search interests for 27 broad domains capturing social life and psychological functioning. To this end, 249 search terms from the Google Trends database were used as observed indicators. Factor scores obtained in time series factor analyses were used to overcome the challenges of unreliability and specificity of single indicator variables. Search interests in all investigated domains were found to be heavily affected by the COVID-19 pandemic. Confirming previous findings, increases in search interests were mostly short-lived [ , ], indicating possible knowledge translation gaps, which could be overcome by adapting public information strategies [ - ].
While disease- and virus-associated domains (ie, pandemic disease, health care institutions, infection, symptoms, hand hygiene, disinfectant, and COVID-19 restrictions) showed early increases in public interest, the domains homeschooling and mask were not triggered by first reports but peaked later, when the lockdown measures had been issued for some time already. Searches for the names of political leaders peaked multiple times, whereas the domains vaccination, social life, hobbies and sports, psychosocial impact, and dispatching revealed more long-term interest. This is in line with previous research, which has shown that people responded to the first reports of infections by seeking out information about COVID-19, as measured by searches for coronavirus, coronavirus symptoms, and hand sanitizer . Contrary, personal health strategies (ie, mask, medication) and community-level policies (ie, school closures, testing) were found to peak at later stages of the pandemic [ ]. Additionally, differences between domains and their delayed occurrence have been reported multiple times [ , , ].
We did not find meaningful differences between German federal states for any of the studied domains. This corresponds with high correlations for the single search term “coronavirus” for multiple countries in previous research . However, other research has shown differences in the onset of COVID-19 search terms in Google Trends, even for subregions within countries [ ]. Hence, we assume that the highly comparable search interests observed across German federal states could reflect, in part, that German federal states issued comparable clusters of NPIs simultaneously [ ].
Generally, cross-correlations between domains were found to be highest for nonshifted time series, which we tentatively interpret as indicating that the announcement of NPIs immediately influences public interest. This potentially reveals a decisive point worth considering when developing information strategies.
Change point analyses indicate level changes in public interest over the lockdown period for all domains. However, there was no conclusive evidence that change points for domains consistently coincide with the onset of the NPIs. This suggests that changes observed in search interests for domains are not directly related with the onset of NPIs. Nevertheless, certain domains characterized by a high cross-correlation (eg, pandemic disease and health care institutions) revealed a comparable pattern of change points (seefor all cross-correlations and and for the time series with change points). Change points may thus be prone to changes within and between time series but are not sensitive enough to coincide with the onset of new infection cases or the onset of NPIs. They could provide circumstantial evidence of co-occurrence of events, but they lack sensitivity to identify temporarily lagged effects.
To shed light on possibly lagged effects, all domains were subject to time series regression. In line with previous evidence [, , ] that search interests for COVID-19–related search tags follow increases in infection rates after 8-14 days, daily new infections were a leading predictor in the current study as well. This confirmed a comparably lagged pattern, especially for the domains pandemic disease, health care institutions, symptoms, and hand hygiene within the aforementioned lead (see ). A rise in new infection cases also predicted search interest for convenience goods, economic impact, and business communication but was related to reduced interest in parenting and childcare and renovation. As the lockdown has been shown to considerably affect social life and people’s routines [ , , ], we chose the two NPI factors capturing regulations pertaining to outdoor activities and social life as additional predictors. Both factors appeared to be differentially related with domains. The factor capturing regulation of outdoor activities was positively related with search interests for convenience goods, COVID-19 relaxations, and online shopping. The factor capturing regulation of social activities predicted search interests with the same direction for the same domains as the factor capturing regulation of outdoor activities, and additionally health care institutions, infection, vaccination, social life, hobbies and sports, and psychosocial impact. Further, it was negatively related with the domains symptoms and information seeking. Overall, our results indicate that NPIs targeting the regulation of social life may have more impact on people’s behavior and routines.
The proxy variable for the state of economy (TTMI) revealed relatively consistent negative relations with the investigated search interests of arguably low magnitude. The direction of this effect is plausible, as it indicates increasing interest as the economy declines. However, the small magnitude of these effects suggests that the economic situation had a rather small impact on people’s social and psychological interests in the first lockdown period.
This study used the information available in public databases and Google Trends as a proxy for people’s psychological interests, motives, and concerns. Given that inferences were drawn only from individuals with access to the internet who use the Google search engine, results could be biased to some extent (eg, because older adults or low-income people could be underrepresented in the sample). However, we deem this effect to be rather small, as 95% of households in Germany have access to the internet . Further, there is evidence showing that information drawn from different search engines and social media platforms is highly comparable [ ], hence, results can be expected not to be specific for the Google search engine only.
We investigated whether changes in search interests were affected by the onset of the pandemic and the corresponding lockdown in Germany. However, in a globally connected world, news from abroad could have easily affected search interests. In fact, search interest in COVID-19 reached its first peak before infection rates rose in Europe, especially when infection rates rose exponentially in China , suggesting a global influence [ ]. The fact that regions with the highest numbers of cases were not always the first in terms of spreading information suggests a considerable role of cross-effects [ ]. Consequently, regional, national, and even global events can exert an effect on public interest [ ].
Some of the used search tags (eg, diarrhea) may not be uniquely related to the COVID-19 outbreak . We addressed this challenge in two ways. First, we investigated relatively long baseline phases to test if interest after the arrival of the pandemic increased meaningfully compared to its baseline. Second, we did not base our analyses on single search tags that could easily be affected by third factors. Rather, we estimated interest in domains based on the aggregated information across several search tags.
Regression analysis revealed the necessity to account for correlated residuals during the linear modelling of Google Trends data. Another alternative for time series data would be the usage of vector autoregressive models (VAR) for prediction. VAR offers the advantage that criterion and predictor variables are interchangeable, allowing the use of all variables without prior assumptions about causality. This seems appropriate as cross-correlation functions inshow moderate to high correlations between all studied time series. Even predicting new infection hot spots seems possible by analyzing public interest, given a proper sample size. Although this might be counterintuitive, there is supporting evidence that Google Trends data can be used to find new symptoms caused by diseases [ ].
Although we z-standardized all variables prior to regression analysis, some betas exceeded the value of 1 (see, predictor NPI Social for the domains vaccination and hobbies and sports), possibly indicating suppression effects. However, post hoc correlation analysis between predictors indicated only moderate positive correlation. Thus, predictors are not highly collinear and betas exceeding 1 may reflect computational imprecision.
In line with our preregistration , our study was multivariate and multiple analyses had to be conducted. However, we did not correct statistical significances for multiple testing as we believe that effect sizes of the observed relations are more meaningful and closely in line with previous research for empirically derived domains. Nonetheless, we consider replication in independent samples very valuable to corroborate the obtained results, especially for the rationally selected domains used in the exploratory analyses.
This study shows that people’s search interests are considerably affected by the pandemic and the issued NPIs. Increases were observed across a broad set of domains, reflecting psychological interests, motives, and concerns. For disease- and virus-associated domains, the increases occurred shortly after the onset of the pandemic, whereas domains capturing personal health strategies or community-level policies peaked later during the lockdown period. Renovation was the only domain to show an early decrease in public interest. The different onsets of the studied domains may indicate that some consequences are yet to become apparent, (eg, an increase in mental disorders due to novel life circumstances during the lockdown). Further research could, therefore, tap into multiple, more detailed domains using the demonstrated approach of TSFA while combining multiple keywords to handle undesirable error variance.
Using Google Trends data provides insights into people’s search interests. The high sensitivity backs the interpretation of changes in public interest as a behavioral tendency [, ]. Analyzing Google Trends data solves the problem of data scarcity, providing inexpensive information regarding possible behavioral changes that would be difficult to attain by means of questionnaire and survey data. This study shows pronounced changes in people’s search interests during crises and lockdown, although prediction of specific developments is moderate at best. Given the relatively broad effects across the diverse domains, manifold effects on psychological interests, motives, and worries are indicated. In turn, these effects can help policy makers to develop a better insight into people’s needs and concerns and to adapt and develop suitable information, prevention, and intervention strategies.
DR, PD, and FS contributed to conceptualization and methodology. DR performed data curation and formal analysis, as well as original draft preparation. DR, PD, and FS contributed to manuscript review and editing. We also thank Deniz Dericioğlu for his help in retrieving the keywords for our analysis.
Conflicts of Interest
Overview of domains and corresponding search tags. Search tags we could not retrieve via Google Trends are marked with a *. Search tags that were dropped during data reduction and preprocessing are marked with a †.DOCX File , 43 KB
Map showing the 16 German federal states colored by 7-day incidence of COVID-19 per 100,000 inhabitants for the middle (April 30, 2020) and the end of the lockdown (June 26, 2020) in Germany.DOCX File , 130 KB
Summary statistics for population number and density, as well as cumulative infection and incidence rates for all federal states and Germany.DOCX File , 22 KB
Scree plots for time series factor analysis for each domain after deleting items with negative loadings, in line with the preregistered protocol. All scree plots support the predicted underlying one-factor structure, indicating the communality of the respective domain.DOCX File , 332 KB
Cross-correlation functions for each predictor for leads and lags of up to ±21 days for all domains used in the analysis. Predictive value as a function of lead/lag is given for infection rates, the countermeasure factors capturing regulation of outdoor activities (CM Outdoor) and regulation of social life (CM Social), and the truck toll mileage index as an indicator of the state of economy in Germany.DOCX File , 896 KB
- WHO announces COVID-19 outbreak a pandemic. World Health Organization. 2020 Mar 12. URL: https://www.euro.who.int/en/health-topics/health-emergencies/coronavirus-covid-19/news/news/2020/3/who-announces-covid-19-outbreak-a-pandemic [accessed 2020-10-04]
- Naming the coronavirus disease (COVID-19) and the virus that causes it. World Health Organization. URL: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/technical-guidance/naming-the-coronavirus-disease-(covid-2019)-and-the-virus-that-causes-it [accessed 2020-10-04]
- Robert Koch-Institut: COVID-19-Dashboard. Robert Koch-Institut. URL: https://experience.arcgis.com/experience/478220a4c454480e823b17327b2bf1d4 [accessed 2020-10-04]
- WHO Coronavirus Disease (COVID-19) Dashboard. World Health Organization. URL: https://covid19.who.int/ [accessed 2020-10-04]
- Steinmetz H, Batzdorfer V, Bosnjak M. The ZPID lockdown measures dataset. ZPID Sci Inf Online 2020;20(1):11. [CrossRef]
- Imai N, Cori A, Dorigatti I, Baguelin M, Donnelly C, Riley S, et al. Report 3: Transmissibility of 2019-nCoV. Imperial College London. 2020 Jan 25. URL: https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-transmissibility-25-01-2020.pdf [accessed 2021-05-19]
- Bhatia S, Imai N, Cuomo-Dannenburg G, Baguelin M, Boonyasiri A, Cori A, et al. Estimating the number of undetected COVID-19 cases among travellers from mainland China. Wellcome Open Res 2020 Jun 15;5:143. [CrossRef]
- Ferguson N, Laydon D, Nedjati GG, Imai N, Ainslie K, Baguelin M, et al. Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. Imperial College London. 2020 Mar 16. URL: https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf [accessed 2021-05-21]
- Flaxman S, Mishra S, Gandy A, Unwin HJT, Mellan TA, Coupland H, Imperial College COVID-19 Response Team, et al. Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe. Nature 2020 Aug 08;584(7820):257-261. [CrossRef] [Medline]
- Ainslie KEC, Walters CE, Fu H, Bhatia S, Wang H, Xi X, et al. Evidence of initial success for China exiting COVID-19 social distancing policy after achieving containment. Wellcome Open Res 2020 Oct 1;5:81 [FREE Full text] [CrossRef] [Medline]
- Mavragani A, Ochoa G. Google Trends in Infodemiology and Infoveillance: Methodology Framework. JMIR Public Health Surveill 2019 May 29;5(2):e13439 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. Infodemiology: tracking flu-related searches on the web for syndromic surveillance. AMIA Annu Symp Proc 2006:244-248 [FREE Full text] [Medline]
- Eysenbach G. Infodemiology and infoveillance tracking online health information and cyberbehavior for public health. Am J Prev Med 2011 May;40(5 Suppl 2):S154-S158. [CrossRef] [Medline]
- Eysenbach G. Infodemiology: The epidemiology of (mis)information. Am J Med 2002 Dec 15;113(9):763-765. [CrossRef] [Medline]
- Arora VS, McKee M, Stuckler D. Google Trends: Opportunities and limitations in health and health policy research. Health Policy 2019 Mar;123(3):338-341. [CrossRef] [Medline]
- Kaleem T, Miller D, Waddle M, Stross W, Miller R, Trifiletti D. Google Search Trends in Oncology and the Impact of Celebrity Cancer Awareness. International Journal of Radiation Oncology, Biology, and Physics 2018 Nov;102(3):e270-e271. [CrossRef]
- Ling R, Lee J. Disease Monitoring and Health Campaign Evaluation Using Google Search Activities for HIV and AIDS, Stroke, Colorectal Cancer, and Marijuana Use in Canada: A Retrospective Observational Study. JMIR Public Health Surveill 2016 Oct 12;2(2):e156 [FREE Full text] [CrossRef] [Medline]
- Hu D, Lou X, Xu Z, Meng N, Xie Q, Zhang M, et al. More effective strategies are required to strengthen public awareness of COVID-19: Evidence from Google Trends. J Glob Health 2020 Jun;10(1):011003 [FREE Full text] [CrossRef] [Medline]
- Rovetta A, Bhagavathula AS. COVID-19-Related Web Search Behaviors and Infodemic Attitudes in Italy: Infodemiological Study. JMIR Public Health Surveill 2020 May 05;6(2):e19374 [FREE Full text] [CrossRef] [Medline]
- Singh L, Bansal S, Bode L, Budak C, Chi G, Kawintiranon K, et al. A first look at COVID-19 information and misinformation sharing on Twitter. ArXiv. Preprint posted online on March 31, 2020 2020 Mar 31. [Medline]
- Rajan A, Sharaf R, Brown R, Sharaiha R, Lebwohl B, Mahadev S. Association of Search Query Interest in Gastrointestinal Symptoms With COVID-19 Diagnosis in the United States: Infodemiology Study. JMIR Public Health Surveill 2020 Jul 17;6(3):e19354 [FREE Full text] [CrossRef] [Medline]
- Abd-Alrazaq A, Alhuwail D, Househ M, Hamdi M, Shah Z. Top Concerns of Tweeters During the COVID-19 Pandemic: Infoveillance Study. J Med Internet Res 2020 Apr 21;22(4):e19016 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. Infodemiology and infoveillance: framework for an emerging set of public health informatics methods to analyze search, communication and publication behavior on the Internet. J Med Internet Res 2009 Mar 27;11(1):e11 [FREE Full text] [CrossRef] [Medline]
- Rivers C, Chretien J, Riley S, Pavlin JA, Woodward A, Brett-Major D, et al. Using "outbreak science" to strengthen the use of models during epidemics. Nat Commun 2019 Jul 15;10(1):3102 [FREE Full text] [CrossRef] [Medline]
- Knipe D, Evans H, Marchant A, Gunnell D, John A. Mapping population mental health concerns related to COVID-19 and the consequences of physical distancing: a Google trends analysis. Wellcome Open Res 2020 Apr 30;5:82. [CrossRef]
- Heymann DL, Shindo N, WHO Scientific and Technical Advisory Group for Infectious Hazards. COVID-19: what is next for public health? Lancet 2020 Feb 22;395(10224):542-545 [FREE Full text] [CrossRef] [Medline]
- Kardeş S, Kuzu AS, Pakhchanian H, Raiker R, Karagülle M. Population-level interest in anti-rheumatic drugs in the COVID-19 era: insights from Google Trends. Clin Rheumatol 2021 May;40(5):2047-2055 [FREE Full text] [CrossRef] [Medline]
- Kardeş S, Kuzu AS, Raiker R, Pakhchanian H, Karagülle M. Public interest in rheumatic diseases and rheumatologist in the United States during the COVID-19 pandemic: evidence from Google Trends. Rheumatol Int 2021 Feb;41(2):329-334 [FREE Full text] [CrossRef] [Medline]
- Strzelecki A. The second worldwide wave of interest in coronavirus since the COVID-19 outbreaks in South Korea, Italy and Iran: A Google Trends study. Brain Behav Immun 2020 Aug;88:950-951 [FREE Full text] [CrossRef] [Medline]
- Bento AI, Nguyen T, Wing C, Lozano-Rojas F, Ahn Y, Simon K. Evidence from internet search data shows information-seeking responses to news of local COVID-19 cases. Proc Natl Acad Sci U S A 2020 May 26;117(21):11220-11222 [FREE Full text] [CrossRef] [Medline]
- Husnayain A, Shim E, Fuad A, Su EC. Understanding the Community Risk Perceptions of the COVID-19 Outbreak in South Korea: Infodemiology Study. J Med Internet Res 2020 Sep 29;22(9):e19788 [FREE Full text] [CrossRef] [Medline]
- Effenberger M, Kronbichler A, Shin JI, Mayer G, Tilg H, Perco P. Association of the COVID-19 pandemic with Internet Search Volumes: A Google Trends Analysis. Int J Infect Dis 2020 Jun;95:192-197 [FREE Full text] [CrossRef] [Medline]
- Walker A, Hopkins C, Surda P. Use of Google Trends to investigate loss-of-smell-related searches during the COVID-19 outbreak. Int Forum Allergy Rhinol 2020 Jul;10(7):839-847 [FREE Full text] [CrossRef] [Medline]
- Sousa-Pinto B, Anto A, Czarlewski W, Anto JM, Fonseca JA, Bousquet J. Assessment of the Impact of Media Coverage on COVID-19-Related Google Trends Data: Infodemiology Study. J Med Internet Res 2020 Aug 10;22(8):e19611 [FREE Full text] [CrossRef] [Medline]
- Cherry G, Rocke J, Chu M, Liu J, Lechner M, Lund VJ, et al. Loss of smell and taste: a new marker of COVID-19? Tracking reduced sense of smell during the coronavirus pandemic using search trends. Expert Rev Anti Infect Ther 2020 Nov;18(11):1165-1170 [FREE Full text] [CrossRef] [Medline]
- Li C, Chen LJ, Chen X, Zhang M, Pang CP, Chen H. Retrospective analysis of the possibility of predicting the COVID-19 outbreak from Internet searches and social media data, China, 2020. Euro Surveill 2020 Mar;25(10):1-5 [FREE Full text] [CrossRef] [Medline]
- Yuan X, Xu J, Hussain S, Wang H, Gao N, Zhang L. Trends and Prediction in Daily New Cases and Deaths of COVID-19 in the United States: An Internet Search-Interest Based Model. Explor Res Hypothesis Med 2020 Apr 18;5(2):1-6 [FREE Full text] [CrossRef] [Medline]
- Lin Y, Chiang T, Lin Y. Increased Internet Searches for Insomnia as an Indicator of Global Mental Health During the COVID-19 Pandemic: Multinational Longitudinal Study. J Med Internet Res 2020 Sep 21;22(9):e22181 [FREE Full text] [CrossRef] [Medline]
- Mavragani A. Tracking COVID-19 in Europe: Infodemiology Approach. JMIR Public Health Surveill 2020 Apr 20;6(2):e18941 [FREE Full text] [CrossRef] [Medline]
- Lin Y, Liu C, Chiu Y. Google searches for the keywords of "wash hands" predict the speed of national spread of COVID-19 outbreak among 21 countries. Brain Behav Immun 2020 Jul;87:30-32 [FREE Full text] [CrossRef] [Medline]
- Ayyoubzadeh SM, Ayyoubzadeh SM, Zahedi H, Ahmadi M, R Niakan Kalhori S. Predicting COVID-19 Incidence Through Analysis of Google Trends Data in Iran: Data Mining and Deep Learning Pilot Study. JMIR Public Health Surveill 2020 Apr 14;6(2):e18828 [FREE Full text] [CrossRef] [Medline]
- Brunori P, Resce G. Searching for the Peak Google Trends and the COVID-19 Outbreak in Italy. SSRN Journal 2020:1-13. [CrossRef]
- Liu D, Clemente L, Poirier C, Ding X, Chinazzi M, Davis J, et al. Real-Time Forecasting of the COVID-19 Outbreak in Chinese Provinces: Machine Learning Approach Using Novel Digital Data and Estimates From Mechanistic Models. J Med Internet Res 2020 Aug 17;22(8):e20285 [FREE Full text] [CrossRef] [Medline]
- Cousins HC, Cousins CC, Harris A, Pasquale LR. Regional Infoveillance of COVID-19 Case Rates: Analysis of Search-Engine Query Patterns. J Med Internet Res 2020 Jul 30;22(7):e19483 [FREE Full text] [CrossRef] [Medline]
- Gane SB, Kelly C, Hopkins C. Isolated sudden onset anosmia in COVID-19 infection. A novel syndrome? Rhinology 2020 Jun 01;58(3):299-301. [CrossRef] [Medline]
- Goldman D. Initial Observations of Psychological and Behavioral Effects of COVID-19 in the United States, Using Google Trends Data. SocArXiv 2020 Apr 04:1-5. [CrossRef]
- Brodeur A, Clark AE, Fleche S, Powdthavee N. COVID-19, lockdowns and well-being: Evidence from Google Trends. J Public Econ 2021 Jan;193:104346 [FREE Full text] [CrossRef] [Medline]
- Schiller F, Bey K, Grabe L, Schmitz F. Stockpile purchasing in the emerging COVID-19 pandemic is related to obsessive-compulsiveness. Journal of Affective Disorders Reports 2021 Apr;4:100116. [CrossRef]
- Rotter D, Doebler P, Schmitz F. Information search in times of crisis: How Google trends can inform policy makers. OSF Registries. 2020. URL: https://osf.io/s5379 [accessed 2021-05-21]
- Massicotte P, Eddelbuettel D. gtrendsR: Perform and Display Google Trends Queries. The Comprehensive R Archive Network. 2020. URL: https://cran.r-project.org/package=gtrendsR [accessed 2021-05-19]
- R: A Language and Environment for Statistical Computing. R Core Team. 2017. URL: https://www.r-project.org/ [accessed 2021-05-19]
- Situationsbericht vom 28.4.2020. Robert Koch Institut. 2020 Apr 28. URL: https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Situationsberichte/2020-04-28-de.pdf [accessed 2021-05-19]
- Lkw-Maut-Fahrleistungsindex. Statistisches Bundesamt. URL: https://www.destatis.de/DE/Service/EXDAT/Datensaetze/lkw-maut-artikel.html [accessed 2020-10-04]
- Gilbert P, Meijer E. Time series factor analysis with an application to measuring money. University of Groningen. 2005. URL: https://research.rug.nl/en/publications/time-series-factor-analysis-with-an-application-to-measuring-mone-2 [accessed 2021-05-21]
- Rossiter D. Tutorial: Time series analysis in R. URL: http://www.css.cornell.edu/faculty/dgr2/_static/files/R_PDF/exTSA.pdf [accessed 2020-07-03]
- Zeileis A, Leisch F, Hornik K, Kleiber C. strucchange: An R Package for Testing for Structural Change in Linear Regression Models. J Stat Soft 2002;7(2):1. [CrossRef]
- Box GEP, Jenkins GM, Reinsel GC, Ljung GM. Time series analysis: forecasting and control. Hoboken, NJ: John Wiley & Sons; 2015.
- Schwarz G. Estimating the Dimension of a Model. Ann Statist 1978 Mar 1;6(2):461-464. [CrossRef]
- Bates D, Mächler M, Bolker B, Walker S. Fitting Linear Mixed-Effects Models Using lme4. J Stat Soft 2015;67(1):1. [CrossRef]
- Share of households with internet access in Germany from 2007-2019. Statista GmbH. URL: https://www.statista.com/statistics/377677/household-internet-access-in-germany/ [accessed 2020-10-04]
|API: advanced programming interface|
|ARIMA: autoregressive integrated moving average|
|ICC: intraclass correlation|
|NPI: nonpharmaceutical intervention|
|NPI Social: NPI factor regulating social life|
|NPI Outdoor: NPI factor regulating outdoor activities|
|RKI: Robert Koch Institute|
|TSFA: time series factor analysis|
|TTMI: truck toll mileage index|
|VAR: vector autoregressive models|
|WHO: World Health Organization|
|ZPID: Leibniz Institute for Psychology|
Edited by C Basch; submitted 09.12.20; peer-reviewed by A Mavragani, S Kardes; comments to author 13.01.21; revised version received 26.01.21; accepted 15.04.21; published 01.06.21Copyright
©Dominik Rotter, Philipp Doebler, Florian Schmitz. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 01.06.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.