This study employed a quantitative cross-sectional design to evaluate the reliability and biases of SMPs compared to conventional epidemiological methods for collecting health-related data. We used Twitter and Mastodon, a decentralized social media platform, to gather data on COVID-19 infections, integrating this approach with a linked online LimeSurvey questionnaire. Additionally, data from conventional sources, including the MuSPAD and COSMO studies, as well as official reports from the Robert Koch Institute (RKI) and Federal Statistical Office of Germany (FSO), were used for comparison.

Our study was preregistered on OSF in July 2023 [48]. Our third preregistered hypothesis focused on the derivation of contact network properties (H3: During the COVID-19 pandemic, individuals exhibited more pronounced social selectivity based on a homophily—the tendency to associate with similar others—due to increased awareness of their “second-order contacts” [contacts of contacts]) is tested in a second, forthcoming publication [49]. The recruitment strategy was developed afterward to increase reach and was introduced during the survey development. Bootstrapping was also introduced at a later point to address recruitment bias.

This study was conducted in accordance with the Declaration of Helsinki. Informed consent was obtained from all participants. It was approved by the ethics committee of the University of Lübeck (approval number 2023-413). Reporting was conducted in accordance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) reporting guidelines [50]. Recruiters were not compensated in any way and included study authors, as well as one person affiliated with the study institutions. Sharing the recruitment link via social media merely served as a neutral means of participant recruitment and did not provide the authors with any personal, financial, or competitive advantage influencing the study’s outcomes.

Clarification and obtaining consent to share nonpersonal data as part of a social media survey took place in terms of use of the respective social media services. Apart from the publicly accessible survey data, no other, and thus, in particular, no personal data are used.

We enlisted 5 German-speaking recruiters, each with an established social media presence, to distribute the SMPs. All recruiters were informed about the study and willingly agreed to participate. The recruiters had follower counts ranging from approximately 750 to 65,000 on Twitter (Table 1). Recruiter 1, who also had a Mastodon account with over 8300 followers, posted identical polls on both platforms, enabling platform comparisons. At the time, Recruiter 1 was the only recruiter with more than 8300 followers on Mastodon, which we deemed necessary for a reliable comparison of results.

Polls were posted from July 19 to July 26, 2023 (7 days being the maximum due to platform limits) and included 2 multiple-choice questions on COVID-19 infection history. The polls were structured to allow respondents to select their answers or view results without participating.

A custom-built Twitter posting tool automated the posting process for 3 recruiters, ensuring consistent timing across accounts. Recruiters who posted manually followed similar guidelines to minimize timing bias. The polls were posted in a thread, with the last post being a post with a link to an external LimeSurvey questionnaire, allowing respondents to transition seamlessly from poll participation to survey completion. From now on, external survey always refers to the externally hosted LimeSurvey questionnaire that was linked at the end of the Twitter or Mastodon thread. All questions were originally formulated in German and translated for the purpose of this paper (see also Table S1 in Multimedia Appendix 1).

The initial social media post briefly explained the study purpose to ensure transparency and provide context. An anonymized example of the Twitter thread is shown in Figure 1, where identifiable details, including the location of the team and the survey link, have been removed.

The MuSPAD data were merged from the independent waves using the following procedure: use the “user_id” column to assign results to participants and merge all results. MuSPAD collected the number of infections up to the end of 2022; participants could only report 1 date of infection for the period from April 1, 2023, to August 31, 2023. Therefore, incidence during this period was calculated based on a maximum of 1 reported infection per participant.

We apply weighted bootstrapping to ensure that the age distribution of participants in the external survey matches the age distribution of the German population (see Demographic Comparison section). This also allows for comparisons of the 7-day incidence per 100,000 between the external survey, the MuSPAD study, and the official reporting statistics by RKI. To compute the mean 7-day incidence and the empirical 95% CI of this mean for each point in time, we apply bootstrapping. We perform 1000 bootstrap iterations, resampling with replacement within each age group, with sample sizes weighted according to population age distributions, to generate incidence estimates.

We evaluated the representativeness and reliability of the data collected via SMPs and the external survey against the following conventional sources:

For demographic comparisons, gender [57], age [58], household size [59], education level [60], and occupation [61] data were used, sourced from MuSPAD and national statistics. Age brackets were introduced into the data to facilitate comparison with other data. These consist of the age-in-years brackets: 18‐39, 40‐59, 60‐79, and 80‐99.

Descriptive statistics were used to analyze the distribution of infection rates, vaccination history, and demographic data. Comparisons included differences between Twitter and Mastodon responses, as well as variations among the 5 recruiters. Furthermore, we excluded the “show results” votes on the SMPs from the analysis (“show results” means that on Twitter and Mastodon, users submitted no valid answer themselves but only looked at the votes of others, providing no data points).

To estimate differences in infection distributions, we used contingency tables and reported Cohen w as a measure of total sample differences. To compare infection timelines across data sources, we calculated pairwise Pearson and Spearman correlation coefficients and conducted corresponding correlation tests to obtain P values for each comparison. Data from the external survey were compared against MuSPAD and the FSO to quantify bias.

We computed 95% CIs for any sample proportion, p^. As we assumed a binomial distribution for the true population proportion and as the binomial distribution is approximately normal for large enough samples, we used z scores when computing the CIs.

We found similar results for all samples regarding infection frequency, timing of infections, number of vaccinations, 7-day incidences, and demographic differences, although some differences could be explained by different measurement time periods or differences in measurement methods. Based on a sample of 6127 responses on Twitter and Mastodon and 867 responses from the external survey, the self-reported frequency of infection aligned closely with conventional sources, such as MuSPAD. Across all 4 studies, 28% to 38% of respondents reported never being infected (Twitter 28%, Mastodon 38%, external survey 34%, MuSPAD 33%), 50% to 56% reported one infection (Twitter 55%, Mastodon 50%, external 56%, MuSPAD 56%), and 8% to 17% reported 2 or more infections (Twitter 17%, Mastodon 12%, external 10%, MuSPAD 12%). The differences compared to MuSPAD were significant but showed only small effect sizes (Cohen w for Twitter: 0.152, Mastodon: 0.188, and external survey: 0.105). We also compared the subsample of SMPs by recruiters, indicating that follower size did not strongly impact findings.

Table 3 provides an overview of the demographic composition and self-reported COVID-19 infection history across the different samples. While the age distributions of Twitter and Mastodon participants are comparable, both differ from the MuSPAD dataset, which includes a broader age range. The proportion of self-reported infections is also relatively consistent across social media–based samples and MuSPAD, although participants from MuSPAD reported fewer repeat infections. These similarities and differences highlight the role of recruitment methods in shaping sample characteristics.

The results from Twitter, Mastodon, the external survey, and MuSPAD are largely consistent in terms of reported infection history (Figure 2). For these 4 studies, the proportion of respondents who stated that they had never been infected is comparable, varying between 28% and 38% (Twitter: 1191/4217, 28%, 95% CI 26.9%-29.6%; Mastodon: 666/1765, 38%, 95% CI 35.5%-40%; external survey: 294/866, 34%, 95% CI 30.8%-37.1%; MuSPAD: 1475/4997, 30%, 95% CI 28.3%-30.8%). Similarly, around half of the respondents of each study stated that they had been infected once (Twitter: 2310/4217, 55%, 95% CI 53.3%-56.3%; Mastodon: 882/1765, 50%, 95% CI 47.6%-52.3%; external survey: 483/866, 56%, 95% CI 52.5%-59.1%; MuSPAD: 2920/4997, 58%, 95% CI 57.1%-59.8%). The percentage of respondents who reported experiencing at least 2 infections ranged from 8% to 17%, while the highest share was recorded on Twitter at 17% (716/4217, 95% CI 15.8%-18.1%), followed by Mastodon at 12% (216/1765, 95% CI 10.7%-13.8%), the MuSPAD study at 12% (602/4997, 95% CI 11.1%-12.9%), and the external survey at 10% (89/866, 95% CI 8.3%, 12.3%).

Overall, the number of infections reported on Twitter showed a difference compared to the reference MuSPAD data with Cohen w 0.152, Mastodon with Cohen w 0.188, and the external survey with Cohen w 0.105. We additionally computed a sensitivity analysis, which indicated that, given the smallest sample size (n=866) and assuming α=.01 and power=0.95, the study could detect effects of Cohen w of at least 0.154. We opted for a post hoc sensitivity analysis instead of a power analysis, as we are mostly interested in determining the threshold for differences that we could have detected rather than determining the power retrospectively.

In contrast, a comparison with the COSMO study shows visible differences. In the most recent round of the COSMO study, conducted on November 29, 2022, and November 30, 2022, 50% of participants (501/1003) reported that they had never been infected, 42% of participants (421/1003) reported that they had been infected once, and 8% (80/1003) reported that they had been infected at least 2 times. Due to its implementation period, the COSMO study does not account for infections that occurred in 2023, impeding a meaningful comparison between the COSMO study and the other 4 studies. However, higher counts for zero infections seem reasonable at an earlier point in time. Summarizing, we can state that the number of infections was captured similarly between samples.

A comparison of the timing of infections (Figure 3) shows that the 7-day incidence per 100,000 from the external survey, the MuSPAD study, and the officially reported incidence by the RKI follow the same trend from March 2020 until data collection ended in summer 2023 (Twitter/Mastodon questions are excluded from the timing comparison as they do not allow the computation of a 7-day incidence per 100,000 for COVID-19 cases. See the Data Collection section for question formulations and Timing of Infection [Twitter and Mastodon] in the Supplementary for discussion of the votes on question 2.). Specifically, waves, local maxima, and local minima occur simultaneously in all 3 data sources. We further note that the local maxima in July 2022 and October 2022 are larger for the external survey and the MuSPAD study, reaching values around 1500, while the 7-day incidence per 100,000, according to the RKI, only reaches values around 500. Overall, patterns of infections are reasonably similar between samples, although measurements were conducted differently.

Spearman rank correlation analyses demonstrated strong positive associations between infection dates in the external survey and MuSPAD (ρ=0.884, P<.001) and between MuSPAD and RKI counts (ρ=0.831, P<.001). The correlation between the external survey and RKI counts was large as well (ρ=0.640, P<.001). Corresponding Pearson coefficients (0.839, 0.856, and 0.599, respectively) showed similar patterns of association. All correlations were statistically significant, indicating temporal concordance across the 3 data sources.

Both the external survey and the MuSPAD study undersample 1-person households. Since the external survey overrepresents 40- to 59-year-old people (see Demographic Comparison section), we applied bootstrapping to adjust the age distribution for comparability with RKI-reported 7-day incidence per 100,000 (see Data Processing section). As the adjustment had a negligible impact on trends, we present the raw data here, with bootstrapped results available in Bootstrapping in Multimedia Appendix 1.

The analysis of the number of vaccination doses received until September 11, 2023, shows comparable results between the external survey and the MuSPAD study (Figure 4). That is, almost all participants received at least 2 doses of a COVID-19 vaccine of any brand, with no discernible differences between the 18- to 39-year-olds, 40- to 59-year-olds, 60- to 79-year-olds, and 80- to 99-year-olds. However, the MuSPAD study shows slightly lower percentages. In both the external survey and the MuSPAD study, we find a small drop between the share of participants who reported receiving at least 2 doses and those who reported receiving 3 doses for the 18 to 39, 40 to 59, and 60 to 79 years age groups, not distinguishing between homologous or heterologous booster shots. A notable difference between the 2 studies emerges in the share of participants who received at least 4 doses of the COVID-19 vaccine, although fourth dose recommendations were intended for older or at-risk populations. Across all age groups, apart from the 80 to 99 years age group, the external survey reports higher vaccination rates than the MuSPAD study.

The comparison of vaccination numbers between the 2 studies and the officially reported numbers by the RKI reveals 2 key differences. First, the RKI splits the adult population only into 2 age groups (18‐59 and 60+ years), thus limiting the comparability. Second, it can be noted that both the external survey and the MuSPAD study struggle to recruit unvaccinated individuals and individuals who decided not to or could not receive a third or fourth vaccine dose. However, the reported variable COVID-19 vaccination status showed a considerable amount of missing data (300/867, 34.6%), which might include unvaccinated individuals.

Analysis of responses to the first Twitter or Mastodon question reveals that most votes stem from Recruiter 1 and Recruiter 2 (5589/6172, ~90%), making them the dominant contributors to the Twitter sample (Table 4).

Breaking down infection history by recruiter reveals variations in reported infection rates. The shares of the participants who reported zero infections on Twitter, for example, fluctuate between 18% (17/97, Recruiter 5) and 31% (642/2054, Recruiter 1 [Twitter]) (see Figure 5). On Mastodon, however, the share of participants who reported zero infections is higher (666/1765, 38%). A similar, though less pronounced, variation is observed among participants who reported one infection: 50% (882/1765) of participants recruited by Recruiter 1 (Mastodon), 54% (863/1605) of participants recruited by Recruiter 2, 54% (1117/2054) of participants recruited by Recruiter 1 (Twitter), 57% (60/106) of participants recruited by Recruiter 4, 59% (57/97) of participants recruited by Recruiter 5, and 60% (211/353) of participants recruited by Recruiter 3 reported one infection. Recruiter 5 recruited, with 23% (22/98), the largest share of participants who reported at least 2 infections. However, the limited number of responses from Recruiter 5’s poll had little impact on the overall Twitter sample distribution. In summary, all recruiters drew similar samples regarding total infections.

A demographic comparison of the external survey and the MuSPAD study with official statistics reveals notable deviations on various levels. That is, females are slightly overrepresented in the external survey but less so than in the MuSPAD study (external survey: 464/857, 54%, MuSPAD: 5966/9842, 61%, FSO: 51%; Figure 6A). Consequently, males are slightly underrepresented in both studies (external survey: 386/857, 45%, MuSPAD: 3861/9842, 39%, FSO: 49%), while respondents who indicated their gender as diverse make up less than 1% of both study’s samples (the FSO only distinguishes between “female” and “male”). In addition, participants aged 40 to 59 years are substantially overrepresented in the external survey (external survey: 569/854, 67%, MuSPAD 3486/9814, 36%, FSO: 27%). In contrast, individuals aged 80 to 99 years are underrepresented in both the external survey and the MuSPAD study (external survey: 2/854, <1%, MuSPAD: 631/9814, 6%, FSO: 9%; Figure 6B). Both the survey and the MuSPAD study undersample 1-person households (external survey: 199/857, 23%, MuSPAD: 2134/7768, 28%; Figure 6C), which make up 41% of households in Germany according to the FSO. Consequently, larger household sizes are slightly overrepresented. Furthermore, most external survey and MuSPAD participants (external survey: 421/565, 75%, MuSPAD: 6931/8608, 81%) replied that they had no children under the age of 14 years (Figure 6D). One (external survey: 72/565, 13%, MuSPAD: 864/8608, 10%) and 2 (external survey: 60/565, 11%, MuSPAD: 667/8608, 8%) children under the age of 14 years are similarly likely in both studies, while only 2% (external survey: 12/565, 2%, MuSPAD: 146/8608, 2%) of respondents reported having 3 or more children under the age of 14 years. Participants who have received higher education are overrepresented in the external survey (external survey: 485/565, 86%, MuSPAD: 5085/9679, 52%, FSO: 34%). Thus, only 7% (37/565) reported they had obtained a certification after 10 years (MuSPAD: 2454/9679, 26%, FSO: 30%), and less than 1% (2/565) reported that they had obtained a certificate after 9 years (MuSPAD: 927/9679, 12%, FSO: 29%; Figure 6E). Finally, the external survey oversampled participants who reported their current occupation as “other” (external survey: 407/563, 72%, MuSPAD: 4558/9791, 47%, Federal Employment Agency: 55%), while it undersampled retired participants (external survey: 53/563, 9%, MuSPAD: 3546/9791, 36%, Federal Employment Agency: 30%; ).

To conclude, both novel sampling methods (eg, external survey, SMPs) introduce biases regarding demographic data.

In the time period assessed here—3 years into a coronavirus pandemic and after the establishment of large testing schemes in the population—the data collected via SMPs mirror cumulative self-reported infections in the assessed epidemic panel. In particular, it matches the proportion of people reporting having had infections. This shows the potential of SMPs, which can be an adequate and cost-effective proxy for infection numbers in such situations and can provide rapid and readily available data for dynamic modeling. Epidemic panels are particularly valuable during pandemic situations, as they enable serological estimates of cumulative infection prevalence and real-time monitoring of infection rates, helping to identify potential underreporting of self-reported diagnoses. This applies to both early pandemic phases (before large-scale testing is available) and later intrapandemic periods (after interest in testing has declined), as well as to seasonal infections commonly monitored in epidemic panels, such as influenza, respiratory syncytial virus, or vector-borne diseases, where diagnosed cases often underestimate actual infections, particularly across different age groups.

If the focus is on the cumulative number of infections during mid-intrapandemic periods, near real-time and straightforward data collection methods, such as SMPs and the external survey, can provide rapid, directional insights, showing only small differences compared to larger serological studies like MuSPAD. Incidence gathered through recruitment via social media for an external survey is also strongly correlated with official data from public health authorities. While it is notable that our sample can produce such timely results at minimal cost, claims about reliability and representativeness should be tempered, acknowledging that they are limited to pandemic scenarios with similar disease characteristics such as COVID-19.

On the other hand, demographic comparisons reveal that 40‐ to 59-year-olds and individuals with higher education are overrepresented in the data, while 1-person households are underrepresented. The higher education level is expected because all recruiters are part of academia. Due to legal reasons, participants under 18 years could not participate in the SMP, which is a drawback compared to the official testing conducted by the RKI.

While these sample biases and statistically significant differences in infection numbers across different platforms did exist, effect sizes were small, indicating that SMPs could be useful in ascertaining infection trends in the population in a rapid and cost-effective way.

Similar to what Schober et al [33] suggest, SMPs might be used as additional, cost-effective interim or on-top data collection tools to enrich official data. We go beyond what Vidal-Alaball et al [47] achieved in their study, focusing on collecting data instead of evaluating public opinion. As Zhao et al [36] point out, we did consider bias in our data and explicitly tested for this, observing inherently higher education levels in the Twitter data compared to other survey types. While social media data have previously been used for opinion mining and behavioral studies in public health, SMPs have not been systematically applied to directly collect self-reported infection or incidence data.

In terms of comparison to serological indicators of cumulative infections at certain time periods, the presented data on self-reported infections are in line with published data from the IMMUNEBRIDGE project, of which the MuSPAD cohort was one part [62]. Comparison with COSMO data revealed that, due to the earlier data collection period of COSMO, infection numbers were lower compared to our data and MuSPAD, possibly indicating that a large portion of first infections occurred in 2023.

Data until 2022 showed that survey-based incidence aligned closely with the officially reported figures from the RKI. However, a discrepancy emerged with the onset of the Omicron BA.5 wave in the summer of 2022. This could be due to individuals confirming infections with rapid antigen tests that were not reported or because many experienced only mild symptoms and did not seek medical attention, thus not entering official statistics. SMPs here allow accounting for this decline in official testing. Additionally, a potential limitation was that in the MuSPAD data, reported infections in the time frame April 1, 2023, to August 31, 2023, were limited to a single date, excluding possible secondary infections. Due to this, infection numbers during that time frame might be underreported.

Recall bias might have affected the reported timing of infections. shows that the incidence still aligns quite well with official reporting, indicating that the effect of recall bias was small overall.

Regarding vaccination doses, our data align well with the MuSPAD data except for the fourth vaccination dose. Here, our limited sample size regarding older age groups (60+) might be problematic because the fourth vaccination is recommended by the RKI only for these older age groups. Overall, compared to the RKI data, our sample and the MuSPAD sample generally have a higher proportion of vaccinated individuals. This might result from sampling bias, as very risk-averse and protective people might have a higher tendency to participate in a survey regarding a potentially dangerous infectious disease. We also report a high number of missing data points for the COVID-19 vaccination status, which might include unvaccinated individuals who do not want to report their status due to social desirability effects. The effect being present in the MuSPAD data also shows the difficulty in mitigating these effects even in high-effort, high-cost studies.

For the successful recruitment of a large enough sample size, a sizable followership on social media is required. Also, demographic comparisons have shown that biases exist when recruiters are not diverse regarding age, education, and background. Public health–related SMPs are more feasible in the later stages of a pandemic, when tests are widely available to the public and sufficient sentinel studies have already been conducted.

Overall, SMPs can complement traditional methods by providing real-time insights but cannot replace them due to data quality and representativeness concerns.

Finally, it should be noted that there are limitations to SMPs: one does not obtain a list of the users who participated in the survey, but solely the proportion of responses for each option. Thus, this method cannot record demographic attributes, and subanalysis for specific target groups is impossible. Furthermore, only a subset of the population is active on Twitter, limiting the generalizability of these findings.

Another limitation of our study is the very small number of unvaccinated individuals included in the sample (external survey: n=10, 1.76%; MuSPAD: n=489, 3.63%), although the large number of participants who did not provide an answer here might include unvaccinated individuals.

Using 5 academically skewed accounts for recruitment generated 36,416 total Twitter impressions but introduced selection bias through extreme concentration (2 accounts controlled 87% of reach) and network homophily (followers share similar educated, professionally engaged demographics). Response rates of 17% to 19% indicate high engagement; however, the 56% drop in impressions and the 53% drop in votes between Question 1 and Question 2 reveal limitations in data gathering. Mastodon provided 2540 votes, but no impression data were available due to platform limitations, which prevented the calculation of response rates. This recruitment strategy systematically excludes individuals outside academic networks, thereby limiting the findings to “engaged followers of academic social media accounts” rather than more generalizable populations. Additionally, not all recruiters generated enough responses to detect small effect sizes when comparing them to each other. Although the number of infections shows variation, small sample sizes limit any meaningful analysis here.

Due to the limited data collected via Mastodon, H2 could not be fully answered. The Mastodon data, provided solely by Recruiter 1, included a higher proportion of single-person households compared to the overall sample, which may help explain the greater number of participants reporting no infections.

Future work includes improving the representativeness of SMPs, possibly through targeted outreach via ads on Twitter or Facebook, the inclusion of more platforms for data collection (eg, Threads, Bluesky, Facebook), and generating longitudinal data through weekly sampling via a proposed Twitter poll posting tool. This posting tool automatically posts polls every week or 2 weeks to inquire about infections, offering a highly cost-efficient method and allowing for automatic weekly evaluations. Additionally, to test the reliability of this method, a study with in-person testing could be run in parallel.

Especially for recruitment, SMPs show promising results. We were able to generate a sizable sample, although it must be mentioned that this depends heavily on platform follower numbers. We showed that SMPs are an adequate and cost-effective tool for rapidly collecting health data. While overall demographic representativeness is good, discrepancies in age and education may impact generalizability. Especially for health data—in this case, infection numbers and incidence—data quality is comparable to that of more costly and high-effort panel studies. To respond to pandemic scenarios with similar disease characteristics such as COVID-19, data collected via SMPs can quickly provide accurate enough data to help with modeling efforts.