One basic automatic check to protect against bots is CAPTCHAs, which is based on user actions such as checkbox clicks and image challenges [8]. Newer versions like Google’s reCAPTCHA V3 use sophisticated algorithms to evaluate user interactions based on various criteria, such as typing speed and IP address, which are then scored based on the study’s scoring system and expectations in order to identify possible bots [29]. Honeypot questions are an additional automated defense against bots that can be easily included within surveys alongside CAPTCHAs; these are questions that are designed specifically for bot detection and hidden from actual participants using custom code [8]. Thus, receiving answers to a honeypot question would indicate bot activity.

Another widely used strategy to identify and exclude inattentive respondents is attention checks. The most commonly suggested method is to ask for a specific response, otherwise known as a trap question or instructional manipulation check [30]. For example, participants could be asked to pick the italicized option [31], or a specific choice from a list, eg, please select the option “strongly agree” [32]. Current literature advises attention checks to be used throughout the survey to better identify responders who are giving low-quality responses [30]. The wording and quantity of trap questions should strike a balance between filtering out low-quality responses and ensuring that genuine respondents are not accidentally filtered out [32]. An alternative method is to make use of speed bump questions, which require participants to properly read through the question and use reason to arrive at an answer, filtering out bots or respondents who answer hastily or automatically [33]. Speed bump questions can look like this, “The man couldn’t lift his son because he was so weak. Who was weak, the man or his son?” However, researchers should consider the type of research they are doing before making use of attention checks, as they have been found to increase deliberation, potentially affecting cognitive task performance [34]. On the other hand, other research has found that they do not affect participants’ responses on scales or scale validity [35]. Researchers should thus monitor responses or trial survey questions to see if responses are significantly affected before rolling out their surveys officially.

Another strategy researchers can consider using is to leverage domain-specific knowledge to detect respondents who are not from the correct target audience. This is especially effective in fields where many industry-specific terms are used, and researchers can make use of this by including qualitative response formats [30,31]. For instance, one survey targeting experienced programmers included simple programming tasks and questions as assessments [30], while another survey used specific terminology and jargon as soft-checks to verify expertise [31]. Subsequently, identifying inconsistencies or suspicious responses compared to established norms, such as unlikely health symptoms for the given demographic, can identify unfit respondents [36].

Researchers have also found success in using duplicate questions presented in diverse formats. Researchers have advocated for requesting the same data point in multiple formats [29] or incorporating specific items into questionnaires that essentially ask the same question more than once [30]. For example, one study detected bot-based responses by including duplicate gender identity questions, which would trigger subsequent logic checks and reveal bots that followed code rather than survey logic [31]. Inconsistent responses to duplicate questions serve as indicators of low-quality responses, so researchers should take note of mismatches in answers. Researchers should also disguise these checks by distributing questions across different sections of the questionnaire and phrasing them differently to prevent easy identification.

To ensure the integrity of survey responses, researchers can implement rigorous eligibility screening processes and streamline recruitment procedures. For example, researchers can consider using 2-stage recruitment, in which participants first have a public eligibility screener to assess whether they meet eligibility criteria before a second personalized link is sent to them for the main survey [33]. The eligibility screener can further implement features like CAPTCHAs or browser cookies to prevent duplicate responses [33]. Eligibility screeners could also request contact information so that researchers can contact participants personally to confirm their identity or ask for documents that could confirm information critical to eligibility [37]. In order to prevent individuals from tailoring their responses to meet eligibility criteria, researchers can consider recruiting participants without explicitly disclosing eligibility criteria upfront [29]. Furthermore, having duplicated questions in the eligibility screener and main survey will allow researchers to look for inconsistencies in responses, which is another way to expose suspicious respondents [8]. Researchers can consider requiring participants to go through steps before accessing the survey, such as using specialized links or passwords, but must also ensure that this process is as straightforward and user-friendly as possible; otherwise, it risks increasing dropouts [38].

While incentives can promote higher numbers of survey responses, they also run an increased risk of attracting fraudulent responses [33]. To mitigate such risks, researchers can consider structuring participant reimbursements in a way that reduces the financial rewards of fraudulent responses without undermining overall study participation. For example, framing the incentive as a raffle instead of a guaranteed reimbursement reduced the number of bot responses from 633 to 23 in one particular study [29]. This may be explained by the way in which bots are programmed to seek out guaranteed financial incentive surveys, and thus avoid surveys that include random chance incentives [29]. Furthermore, researchers should appropriately match incentives to study requirements and payment norms in the area, as incentives that were seen as higher than others attracted more fraudulent responses [39]. Lastly, separating the reimbursement form and letting participants be directed to it only after finishing the main survey can help ensure only genuine participants receive the incentive [29].

The way in which researchers derive their sample is key, as it is the basis for the entire study. Probability sampling is not typically used with online surveys, simply because it is difficult to establish a probability sample of the whole population on the internet; the population of interest may not necessarily be online, and even those online may not use the internet frequently enough to be captured [40,41]. However, there are workarounds if a probability sample is required, such as randomly sampling and contacting participation via another method (eg, telephone or mail) and doing the actual survey online, or defining a target population (eg, health care workers in a specific geographical area) and inviting all of them to complete the online survey [40]. While probability sampling often leads to a more representative sample of the whole population, it may not necessarily be the most ideal sampling method, as not all research requires the entire population, and it is difficult to obtain representative numbers of smaller subpopulations [23]. Instead, researchers can consider nonprobability sampling, which is ideal for sampling hard-to-reach or hidden populations, where the members are unknown and make up a small proportion of the whole population, meaning that it is difficult to get a sufficiently large sample through probability sampling [22]. Regardless of whether probability or nonprobability sampling is used, the majority of those selected to be sampled should complete the survey; if not, there is a high potential for the sample to be skewed in an unpredictable direction due to potential selection bias and the introduction of large amounts of sampling error [23].

One of the most common sampling methods used in online surveys is river sampling, which is a nonprobability method based upon convenience sampling. In river sampling, researchers place a survey link somewhere it is likely to be seen by members of the target population, such as a web page or through email [24]. However, while cost-effective and convenient, it suffers from coverage bias, where not every subgroup is represented equally on digital platforms, leading to results being most representative of active users of the chosen platform, and not necessarily the target population. It is a great method to quickly recruit large numbers of respondents, but is particularly vulnerable to attacks by fraudulent respondents due to the easy access to the survey [24]. Despite its limitations, river sampling can be very beneficial if the study is focused on a narrower population, such as undergraduate students at a university [1].

Another common sampling method used in online surveys is commercial panels, which can be either probability-based (eg, KnowledgePanel or AmeriSpeak) or nonprobability-based (eg, Qualtrics Panels and Dynata) [28]. Commercial online panel providers include participants who are prerecruited and have agreed to take part in research and differ based on how their participants are recruited. Probability-based panels tend to rely on “offline” recruitment, such as random digit dialing or address-based sampling to invite participants to join, while non-probability-based panels run on a volunteer or opt-in basis [42]. In general, commercial online panels are an alternative option that offer respondent pools that better mirror the demographic compositions of the general population compared to river sampling, but at a greater monetary cost [24]. Commercial panel providers ensure quality by verifying their respondents beforehand, placing the onus of respondent verification on them rather than the researcher [1]. Furthermore, researchers can impose quotas on specific demographic criteria, forcing participants to better approximate the target population. Researchers must keep in mind that even with quotas, online panel providers may still be unable to fully reach hidden populations, as subpopulations within the participant pool may differ in the frequency of how often they actually access the panel and complete surveys, resulting in results that reflect those who are most online rather than being fully representative of the target population [24,43]. The quality of online commercial panels also depends on how it is managed, and researchers should take care in checking how panel participants are recruited or managed before committing to a specific panel in order to maintain a high level of accuracy in the data collected [30].

Lastly, crowdsourcing platforms such as Amazon’s Mechanical Turk and Prolific serve as non–probability-based online marketplaces where potential participants are “hired” to participate in research, but differ from panel providers in that their participant pools are not necessarily curated [24,44]. This means that researchers need to expend extra effort in screening participants for specific target populations, and larger sample sizes are required in order to sufficiently power analysis [45]. Another benefit is that large sample sizes can be generated quickly compared to other methods. While not always ideal for research that requires a sample that is representative of the general population, researchers should leverage the unique strengths of crowdsourcing platforms when studying nondemographic subpopulations or hard-to-reach subpopulations [46]. Such platforms are suitable for experimental work, allowing researchers to establish relationships and effect sizes between different conceptual variables, even if samples are not representative [47]. Although the conclusions drawn are limited, the information is still useful to begin testing hypotheses in diverse samples, making inferences into subpopulation characteristics, and showing that specific phenomena exist [24,48].

A summary of the sampling platforms and methods for online surveys with their respective potential advantages and disadvantages can be found in Table 1.

One of the primary methods of metadata checks is to flag responses that share identical email addresses with previously enrolled participants, or similar email addresses with slight variations in the order of the letters or numbers [7,8]. One example removed any email address that had numbers exceeding 4 digits, as it is a sign of a bot-generated email [33]. To further ensure the quality of such checks, researchers can let the email addresses be checked for any discrepancies by a third researcher after conducting the previous 2 steps of removal based on protocol or duplicates [29]. Reimbursements should be done after all the checks have been conducted. In the case that a genuine participant was excluded erroneously, the research team’s contacts should be available for the participant to contact.

While IP addresses can be shared among legitimate respondents in communal spaces or households, patterns of identical IP addresses across multiple submissions can indicate fraud. Researchers can consider implementing a feature to flag multiple submissions from the same IP address [33]. However, this alone should not be used as an automatic rejection measure, but considered as an aid in the review of responses for potential data quality issues. Although IP addresses can act as a proxy for the legitimacy of a response, such as the geographic location, it is not completely accurate, as participants can fake their IP address or use the same computer [11].

Furthermore, participants may use a virtual private network (VPN) or a virtual private server to change their IP address [29,31]. The use of VPNs and proxies to mask respondents’ true locations poses a significant challenge in ensuring data validity. Such challenges can be addressed by designing tests to detect discrepancies between the time zone reported by the participant’s browser and the IP address’s inferred time zone [31]. If the time zones differed significantly, the response can then be flagged as potentially suspicious. However, researchers must consider that the average internet user is also more likely to make use of VPNs now, meaning that IP addresses may not necessarily be a signal of a low-quality response [15]. Despite its complications, IP address duplication checks are one of the most common data quality checks [39].

Another aspect of metadata checks is the analysis of survey timing and duration. Rapid survey submissions, where multiple surveys are completed within an unrealistically short timeframe, can often indicate bot activity [6,7,11]. Suspicious submissions can be identified by comparing the start and stop times of surveys, flagging those completed within a minute of each other [8]. Survey duration can also be used as a key indicator of fraudulent responses, as overly rapid completion times are unrealistic for genuine participants [39]. Studies have shown that valid respondents typically spend more time on surveys, with response time distributions skewed to the right on a histogram visualization [31]. Pretests or soft launches can help establish realistic time boundaries for survey completion, further enhancing the detection of improbable durations [30,49].

By incorporating multiple qualitative questions and making at least one a requirement for survey submission, researchers can more easily identify automatic responses [33]. One possible protocol is that a response is seen as identical if it is repeated 100 times or more for single words, 10 times or more for 2-word entries, and 3 times or more for entries of 3 or more words [8]. In one study, researchers found that qualitative survey questions, despite being optional, were instrumental in identifying 88 (13.3%) bot responses through exact duplicate answers unlikely to occur by chance [29]. The effectiveness of qualitative questions in detecting bots is supported by research, which found that bots struggled with open-ended questions requiring a minimum response length, suggesting that incorporating multiple such questions can enhance bot detection [29].

Detecting straight-lining, consistencies, or patterns in the response of a participant across a series of questions is crucial for identifying low-effort or automated responses. Thus, it is important to check for uniform answers across grid questions or obvious patterns, such as sequential responses [8,11]. These patterns can indicate a lack of genuine engagement with the survey content. Similarly, verifying selection patterns in multiple response questions and detecting low differentiation in answers also identifies low-quality responses [38]. Furthermore, detecting inconsistencies in responses helps identify respondents who may not be providing truthful or accurate information [29]. It is important to exclude participants with inconsistent responses, such as mismatched ages and birth dates, as this ensures that survey data accurately represents the target population [11]. For example, one study excluded participants whose answers to questions about sex, gender, and sexuality were contradictory, eg, “I have had insertive vaginal sex with multiple female partners,” yet “none of my partners have vaginas” [11]. Implementing comprehensive checks for straight-lining, response patterns, and internal consistency helps filter out low-quality data, ensuring that the final dataset is reliable and valid.

Fraudulent data can occur through a variety of ways, and thus poses distinct challenges for the prevention and detection of such responses [39]. It is important to consider which methods would be best suited for each study, and using multiple methods in a multipronged approach can help to account for the shortcomings of any single method [8,11]. Researchers can consider taking a dynamic approach with evolving screening protocols based on the type of responses received across the study in order to keep up with ever-changing fraud attempts [38]. In addition, when deciding the threshold required to identify a response as fraudulent or low quality, researchers could assign points to indicators of suspicious activity, which reduces the likelihood of incorrectly excluding valid data [8,33,38]. Having a point system allows specific responses to be flagged for further investigation and establishes a cutoff score for invalid responses [30]. This would also follow best practices in which responses would be removed based on multiple issues rather than a single failure.