We conducted a community-engaged longitudinal measurement study using a web-based survey study design involving electronic administration of 2 questionnaires, approximately 1 week apart, with adults living with Long COVID in Canada, Ireland, the United Kingdom, and the United States [35].

The aim of the Long COVID and Episodic Disability Study is to advance the conceptualization and measurement of episodic disability among adults living with Long COVID to inform clinical practice, research, and policy [35-37]. This paper will focus on the second phase of this study, for which the aim was to describe health challenges experienced among adults living with Long COVID and assess the measurement properties of a self-reported disability questionnaire, the Episodic Disability Questionnaire (EDQ) [35]. This phase of the study involved online recruitment of adults living with Long COVID in Canada, Ireland, the United Kingdom, and the United States and electronic administration (via Qualtrics) of 2 online questionnaires (Time 1 and Time 2) [1,35]. Respondents were asked to indicate their email address in the Time 1 questionnaire. We sent a Qualtrics link to a second online questionnaire (Time 2) to the email addresses provided at Time 1, 1 week after completing Time 1 [1]. The questionnaires were estimated to require 30‐40 minutes and 10‐15 minutes to complete for Times 1 and 2, respectively.

We included adults (aged ≥18 years) living in Canada, Ireland, the United Kingdom, and the United States, who self-identified as living with Long COVID, with access to a computer, tablet, or smartphone and internet [38]. We included individuals with suspected or confirmed acute COVID-19 irrespective of whether they were hospitalized or had a positive or negative SARS-CoV-2 test (polymerase chain reaction, antigen, or antibody) [38].

On January 11, 2024, we launched recruitment of participants through online Long COVID community groups (COVID Long Haulers Support Group Canada [39], Long COVID Advocacy Ireland [40], Long COVID Physio [41], Long Covid Support UK [42], and Patient-Led Research Collaborative [43]). Recruitment materials included a poster with a link or QR code to the Time 1 questionnaire, with contact information for the relevant research coordinator, social media posts, and an email to participants from an earlier phase of the study who agreed to be contacted about future research (Multimedia Appendix 1). Information provided in the recruitment materials included a brief description of the study, the inclusion criteria, and indicated that a token of appreciation (a CAD $40 [CAD $1=US $0.72 as of June 15, 2026]) or equivalent currency, an Amazon electronic gift card) would be offered to participants upon completion of the Time 2 questionnaire (Multimedia Appendix 1). Leaders of community groups circulated the study recruitment materials electronically within their networks on social media (including Facebook [Meta Platforms, Inc], Instagram [Meta Platforms, Inc], and X [X Corp]), via email, and word of mouth [44,45]. The social media pages of these groups, some of which were public (ie, no group membership required to view posts), have up to 25,000 followers. Research coordinators emailed the recruitment poster to participants from an earlier phase of the study who provided their contact information and indicated they were interested in participating in future phases of the study [36,37].

The Long COVID and Episodic Disability Study was approved by the Health Sciences REB at the University of Toronto (protocol 41749) and the Saint James Hospital/Tallaght University Hospital Joint Research Ethics Committee (2024-Mar-34453445). The cover page of the Time 1 questionnaire included information about the purpose and rationale of the study, inclusion criteria, potential medical and social risks of participation, the voluntary nature of participation, the types of questions asked (questionnaires), and participants’ right to withdraw at any time. Both Time 1 and Time 2 questionnaires included an electronic eligibility and consent form. Eligible participants provided informed consent by clicking “I agree to participate in this research study.” Data were deidentified. Participants whose responses met the final validity and quality checks received a CAD $40 (CAD $1=US $0.72 as of June 15, 2026) or equivalent electronic gift card at the end of the study.

We describe the resulting number of responses (defined as having viewed the survey, consented to participate, and completed at least 1 item on the questionnaire) across Waves 1 and 2 and Times 1 and Time 2 questionnaire administrations (Figure 2).

Across both waves, we retained 798 of 4663 (17%) responses for Time 1 and 629 of 1281 (49%) responses for Time 2 using our Code Review (Figure 2). Refer to Table 2 for a detailed breakdown of the number of responses removed based on each rule of the Code Review for Waves 1 and 2. These rules were applied together and were not mutually exclusive, and they may not always indicate fraudulent or bot-generated responses. For instance, many responses in Time 1 were removed because of questionnaire incompleteness, represented by ≤80% completion for Time 1 or ≤60% completion for Time 2.

Our experiences conducting an international community-engaged web-based survey with adults living with Long COVID included substantial disruption caused by fraudulent responses and bots. Strategies implemented to prevent and mitigate the threat of fraudulent responses and bots included immediately closing our survey link and making swift amendments to our study protocol (including limiting our recruitment strategy and incorporating additional validity-check questions in our survey questionnaires). Concurrently, we established a set of rules to identify potentially fraudulent responses. Strategies for identifying fraudulent responses included monitoring response completion times, start and end time stamps, geolocation, and screening for suspicious email address characteristics and duplicates. We retained 17% of responses for Time 1 and 49% of responses for Time 2. While not all responses were removed because of fraudulent activity (many were removed because of incompleteness), we identified and removed a substantive number of responses attributed to suspected fraudulent activity. Furthermore, the 2 time points, 1 week apart, in our study design posed challenges as well as opportunities for preventing and detecting fraudulent responses.

Other research teams are experiencing similar challenges with fraudulent responses and bots when administering web-based surveys [9,13,18,22,24,27,29,53,54]. Although information about bots and web-based surveys is emerging for researchers to draw upon when designing web-based surveys, it is challenging for researchers to remain current given the rapidly evolving technology available to fraudulent actors, particularly in the context of generative AI [28-30,33,53,55-57]. This poses a risk to web-based survey research, as the rapid evolution and sophistication of the technology mean that research may continue to be vulnerable.

We outline lessons learned and recommendations for mitigating fraudulent- and bot-related disruptions and enhancing the integrity of web-based surveys. Our lessons learned can be categorized into (1) survey-design and implementation to reduce and identify fraudulent and bot-generated responses, (2) recruitment strategies to decrease the risk of web-based surveys being disrupted by fraudulent actors and bots, and (3) responding to disruptions caused by fraudulent actors and bots. We provide 11 recommendations (Table 3). We suggest the recommended strategies be used in combination to optimize the ability of research teams to prevent and identify bot-generated responses to web-based questionnaires.

Researchers should consider and explore information security features embedded in survey software that can assist in identifying bots and fraudulent actors before launching web-based questionnaires. For example, a CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) may be added to a web-based questionnaire to assess whether a computer user is a human and rule out less sophisticated bots [22,58]. Unfortunately, this is not a foolproof method, as there is ever-evolving technology to decrypt reCAPTCHA [27,58]. Our questionnaire was corrupted by bots despite the use of a CAPTCHA at initial launch. While survey software also may include settings to prevent multiple responses from the same IP address, IP addresses may be obscured intentionally or unintentionally through the use of virtual private networks [22,27,58]. We were able to use IP addresses and associated data on geolocation to identify potentially fraudulent responses in conjunction with other strategies. Survey software companies, such as Qualtrics [1], also may have a bot-detection feature that analyzes responses and provides a score indicating how likely a response is to have been generated by a human. However, these bot-detection features are often moderately effective, can be costly, and should be used in combination with other strategies to protect the integrity of web-based questionnaires [22,27]. We applied these available features during our Wave 2 relaunch, which may have contributed to the improved integrity of our data.

Researchers may design questionnaire items to identify potential bots and fraudulent actors. For example, honeypots are questions that are programmed to engage bot respondents while being invisible to human respondents [17,22]. We added honeypots to our questionnaires in Wave 2; however, we did not receive any responses to these questions despite having other indicators of bots. Other research teams have had similar experiences, suggesting that most bots are now able to evade honeypots [22,59]. Including open-ended questions in web-based questionnaires provides opportunities to assess for potentially fraudulent responses, as researchers can check for identical or illogical responses to questions [23]. In our study, a research coordinator used this method in our early Human Review efforts for screening. It was time-consuming to apply to such a large volume of responses, and at times, difficult to systematically define what constituted an illogical response. As generative AI improves, it may become even more difficult to discern between human- and bot-generated text responses [15]. Given the vulnerabilities of each individual mitigation strategy, we recommend that these strategies be applied in combination. It is critical for researchers conducting web-based surveys to invest in survey platforms with rigorous and current cybersecurity measures to prevent breaches in survey integrity in the first place, as bot-detection strategies are ever-evolving in response to the evolving technology used to implement cyberattacks.

Our experiences with this study taught us the importance of avoiding broad recruitment on social media, avoiding advertising incentives for completing web-based surveys, and engaging community leads in recruitment to reduce the risk of survey corruption by bots. Social media–based recruitment has advantages, notably the potential to reach demographically and geographically diverse groups at a relatively low cost [10,16]. However, fraudulent scams have become common on social media platforms [5,13,17]. The large volume of fraudulent responses we received during Wave 1 likely was influenced by our use of broad social media recruitment and mention of compensation in recruitment materials [5,17,22,28]. The number of fraudulent responses we received substantially decreased in Wave 2 when we only used private groups on social media (monitored by community leads) for recruitment and removed any mention of compensation from recruitment materials. Although avoiding advertising financial incentives may be a critical step to prevent bot-related disruptions to web-based surveys, it can also lead to less diversity among recruited participants [60]. Future research could explore whether other strategies regarding incentives, such as (but not limited to) lottery-based incentives, are effective at minimizing fraudulent responses while maintaining diversity among recruited participants. However, this may introduce issues related to compensating participants unequally. Engaging community leads in our recruitment efforts also helped us target our recruitment to real adults living with Long COVID, as community leads had established relationships within this population and were able to vet potential respondents.

Despite our preventative efforts to reduce fraudulent responses, we experienced an unprecedented influx of responses to our Time 1 questionnaire. Bots will respond to surveys quickly and repeatedly once they have access to a link [22]. When compromised, it is important to close links to web-based questionnaires as quickly as possible once they have been disrupted by bots [11,22]. Despite our relatively quick response (<24 h), we were left with thousands of responses to sort through, muddying the picture of which responses were valid.

We learned several strategies (both manual and automated) for screening data to identify bots and other potentially fraudulent responses. We adopted manual (Human Review) strategies, such as reviewing responses for suspicious names and email addresses, dates of survey completion, and duration of survey completion, that were similar to the approaches implemented by other research teams (Table 3) [13,22]. Manually screening for suspicious patterns in email addresses has been reported by other teams as one of the most effective [13,22], though time- and resource-intensive [24], strategies for identifying bots. A unique aspect of our approach was that we developed code in R (Code Review) to replicate our Human Review [52], while adding new automatic screening criteria. This allowed us to apply our screening criteria efficiently and in a standardized way across all responses we received. A combination of both manual and automatic screening is essential for identifying and removing bot-generated data [13,18,22,24,34].

When responding to disruptions caused by fraudulent and bot-generated responses, it is important to consider both the sensitivity (correctly detecting fraudulent responses) and specificity (correctly identifying true human respondents) of the methodology. It is difficult to achieve high levels of both sensitivity and specificity at the same time, as methods that improve one may reduce the performance of the other. Researchers should navigate the balance between the two, using strategies such as question design, checking survey completion times, and analyzing patterns in responses to classify fraudulent and true responses. In this study, we prioritized sensitivity. This increased our risk of removing true participant responses from the dataset. However, this was necessary to maximize the integrity of our dataset. Regular meetings with the community leads were invaluable for guiding our approach.

Given that fraudulent- and bot-related disruptions to web-based surveys are common among research teams, it is important to document and communicate strategies to prevent and mitigate fraudulent responses [9,13,18,22,24,29]. Strengths of our study included our documented methodical approach to identifying and handling bots and other potentially fraudulent responses. We systematically identified fraudulent responses using a rigorously developed set of automated rules (Code Review). Future research teams using web-based surveys may apply similar strategies. Another strength was our community-engaged approach to participant recruitment. The 5 community groups involved in this study were instrumental in troubleshooting our recruitment strategy, helping us shift to more targeted recruitment methods such as word of mouth, listservs, and private social media groups while still being able to reach our target population of adults living with Long COVID [61].

This web-based survey involved 2 time points of questionnaire administration, 1 week apart. Therefore, when the Wave 1 (Time 1) questionnaire was disrupted by bots, we had to act fast to determine who to invite to complete the Wave 1 (Time 2) questionnaire with the original link (and screen out potentially fraudulent responses). Since the Human Review was manual and time-consuming, we iteratively revised our screening process several times before we developed our Code Review. By that time, we had already sent invitations to some likely fraudulent actors, as indicated by a greater number of Time 2 responses than invitations sent in Wave 1, and screened out some likely legitimate participants (Figure 2). As a result, we created a new link to access the Wave 1 (Time 2) questionnaire to capture as many of our legitimate participants as possible. In other ways, having the second time point to our web-based survey was valuable, allowing us to cross-reference responses between Time 1 and Time 2 to check the quality and effectiveness of our screening approaches (eg, consistency of country of residence between Time 1 and Time 2).

Although our findings are specific to our experiences with a 2-time point survey among people living with Long COVID, the lessons learned and recommendations from this work may be applicable to researchers conducting international community-engaged web-based surveys across other conditions. Strategies to prevent and mitigate fraudulent and bot-generated responses should be tailored to the intended research population using community-engaged approaches throughout. For example, we decided to avoid the use of logic-check strategies given the high prevalence of cognitive challenges among people living with Long COVID [62,63]. For other populations, logic-check strategies may be useful and effective for identifying bots. The recommendations from this work build upon the recommendations of other teams and existing guidelines [18-20,22,23,26,28,29,34,51].

It is likely impossible to prevent all risk of disruption by fraudulent actors and bots when using a web-based survey research design. However, best efforts should be made to reduce this risk and protect the integrity of data collected through web-based questionnaires. It is essential for computer science researchers and coders to develop effective and affordable anti-bot or anti-AI technology that can help protect web-based questionnaires from these threats. Researchers should regularly evaluate their bot-prevention and screening strategies throughout the study design, data collection, and screening. Researchers should also transparently report their process for ensuring data integrity when publishing results from web-based surveys and remain in close correspondence with the relevant REB throughout. Peer reviewers and journal editors should also consider whether and how researchers using web-based surveys have reported their methods for mitigating fraudulent responses during the peer-review process. In addition to threatening data integrity, bot-related disruption may influence the experiences of legitimate research participants. For example, in our study, we were at times delayed in sending invitations to the Time 2 questionnaires, as outlined in our study protocol, and in administering the token of appreciation to participants. It is possible that these disruptions could cause stress among participants or decrease their trust in researchers or community leads involved in recruitment. Therefore, researchers should address risks associated with bot infiltration and bot-screening processes with participants as part of informed consent processes.

Despite the benefits of web-based survey study designs, the risk of fraudulent responses threatens the integrity of online research. Prevention and detection of fraudulent responses may become increasingly difficult because of bot- and AI-assisted fraud. Researchers should proactively plan for the prevention and detection of fraudulent responses, including bots. Their strategies should be well-documented and reported so that researchers can learn from one another and adapt their strategies to keep pace with technological advances in bots and AI. The strategies we have documented and recommended here may be useful to other research communities, but are also specific to our study and experiences. We recommend that research teams implement multiple bot-prevention and detection strategies that are systematic and suited to each study design and study population.