This prospective cohort study collected 1-year retrospective data via interview and prospective data via survey every 2 weeks for 1 year. To understand the flow of participants through study procedures, please see the steps of study enrollment for each participant in Figure 1. We planned to recruit a final sample of 1000 people based on previous online studies in which we were successful in recruiting large samples [56-58].

To characterize the sample, frequency analysis was performed, and descriptive statistics were calculated for participant demographics. We also provide the number and percentage of participants who made it to each step in the enrollment process, as well as the reasons for which participants were excluded from the final sample. Percentages of prospective surveys completed were calculated and displayed in a bar chart.

The CONSORT (Consolidated Standards of Reporting Trials) flowchart (Figure 2) describes how we obtained our final sample of 485 participants who met all eligibility criteria and completed the enrollment procedures with sufficient Google Search data. A summary of our iterations of enrollment procedures is described below. Specific details of each iteration and its results are provided in Multimedia Appendices 3 and 4, respectively.

At the start of the study (March 10, 2022), we asked participants to donate Google Search data on their own using gTAP. To maximize participation, we tried this both prior to and following the completion of the baseline survey. Of the participants in the download screening step asked to download their data on their own (first 4 procedure iterations, n=660), 76.7% (506/660) were lost because they could not donate data on their own (n=278) or lost to follow-up in the process (n=228). Because of the challenges participants reported while attempting to donate data on their own, we changed the procedures to have members of the study team guide participants through their Google data donation at the beginning of their Zoom interview (November 28, 2022). By providing real-time assistance, we lost only 88 participants in this iteration at baseline, mostly due to participant dropout. Although 1975 participants advanced to the download screening step, 325 participants could not donate Google Search data or had insufficient data, and 1063 failed to complete this assessment phase.

A review of potential fraud by research staff led to the exclusion of 272 participants due to 1 or more indicators of fraud. At the Zoom interview stage, 489 participants were eligible to participate based on all screenings. Four participants did not complete the interview, resulting in a final sample of 485 participants who met all inclusion criteria and completed all enrollment steps. Thus, we were not able to recruit 1000 participants as planned.

Table 2 provides the characteristics of the final sample. The participants were 30.3 (SD 10.8) years on average and racially diverse, but there was a disproportionately higher percentage of women. More than half (268/485, 55.3%) of the participants identified as being LGBTQ+ (lesbian, gay, bisexual, transgender, queer, and other). We did not specifically attempt to oversample the LGBTQ+ community; however, we hypothesize that more people felt comfortable disclosing their sexual orientation and/or gender identity because our sample skewed younger and the study was conducted during a time when LGBTQ+ acceptance was high in the United States. Figure 3 displays the geographical representativeness of our sample, which included 47 of the 50 states plus Washington DC. This sample of 485 individuals, all of whom endorsed current suicidal ideation, included 95 (19.6%) participants who attempted suicide in the past year, 285 (58.8%) participants who attempted suicide more than a year ago, and 83 (17.1%) participants with no history of a suicide attempt.

Participant attrition from baseline to week 52 was relatively low. As shown in Figure 4, the completion rates of the prospective surveys remained at 70% or higher across all time points. About three-quarters (71.8%, 348/485) of the fully enrolled participants successfully donated their Google data at week 52. Notably, 96.3% (335/485) of the participants who successfully donated their data had Google data spanning the full prospective year.

There were 124 suicide attempts (by 81 participants) reported during the prospective year. As suicide attempts were expected for our sample, they were not considered to be adverse events in this study by the IRB or DSMB. To our knowledge, 1 participant died, where the cause of death could not be determined (there being no indication that it was suicide). One participant threatened suicide if they were not enrolled in the study. This was addressed by the coprincipal investigator (KAC), a licensed clinical psychologist, who spoke with the participant and determined that they were in active treatment, had local resources available to them, and agreed to call 988, so no further intervention by the study was required.

A leading cause of our inability to meet our intended sample size was participant dropout at different points in the enrollment process. A total of 63.7% (9636/15,120) of the prescreening sample refused to provide consent or dropped out of the study, and 32.3% (4886/15,120) never consented (ie, refused, failed the consent quiz, or were lost to follow-up at this stage). There was less loss to follow-up during the screening (78/4912, 9.7%) and baseline survey phases (74/3366, 2.2%), and only 4 of the 489 individuals (0.8%) who completed the Zoom interview phase were lost to follow-up.

The second leading cause of recruitment attrition was a lack of sufficient Google Search data. Due to a difficulty with donating data, half of the participants (1818/3277, 55.5%) stopped interacting with the study. There were also issues with the download process, resulting in 13.6% (446/3277) of those in the download screening phase being unable to download their data. Moreover, 252 of the 3277 individuals (7.7%) were able to download their data but did not have sufficient searches to qualify for the study (ie, at least 1 Google Search per week for over half the weeks in the previous year).

The third reason for recruitment attrition was fraudulent actors. Of the 15,120 REDCap records at prescreening (the total initial sample), 489 (3.2%) were identified as fraudulent. Just under half (217/489, 44.4%) of the fraudulent actors was identified using automatic checks in REDCap, and the identification of the remainder required research staff time (272/489, 55.6%). The most prominent type of fraudulent acts found was that people were identified as being outside of the United States (273/489, 55.8%). Another common type of fraud was people enrolling in the study multiple times using the same contact information (67/489, 13.7%). Another commonly encountered type of fraudulent acts was identified when the responses given by a participant did not align with their previous responses (75/489, 15.3%).

This study describes the methodological issues involved in conducting an entirely online study involving the download of Google Search data. Our goal was to enroll 1000 United States–dwelling and English-speaking adults experiencing suicidal thoughts or behaviors. The study was able to enroll 485 participants, and the final sample retention was high. The demographics of this sample were typical of remote, longitudinal studies of this nature [58]. We did encounter recruitment challenges owing to the need to screen fraudulent participation and valid participants experiencing difficulty downloading Google Search data or having insufficient amount of Google data to warrant participation. Despite our recruitment challenges, the final sample retention was high, and for participants who could provide data, the data quality was good; however, changes in policies at Google often interfered with the ability to download data. We discuss the specifics of these findings as lessons learned to share with other investigators who wish to pursue research of this nature [56,85,91-98].

Past remote clinical studies have demonstrated an ability to recruit very large samples that are nationally representative and have been able to quickly answer pertinent mental health questions [56,85,95,96]. Indeed, since COVID-19, the number of remote studies has grown considerably [97,98], including those on suicide research [91-93]. In this study, we demonstrated that it is possible to recruit a sample of people at varying risk for suicide from across the United States. Unlike previous remote trials [56,85,95,96], the ability to recruit a large sample, as was originally intended, was very challenging, and while many people attempted to enroll in our study, some were not who they said they were (lesson 2 below) or were unable to donate search data (lesson 3 below).

Our enrolled sample was much less prone to attrition during follow-up than other remote clinical studies [56,99], likely due to successful navigation of the baseline process, automated survey invitations in REDCap, a larger incentive (US $25 instead of US $5) for the longer final survey, and additional tracking and contacts by the study team to maximize completed data donations. However, this only represents 3% of the initial sample and 14% of those who passed screening on suicidality. As the most effective alternative of informing participants about the inclusion criteria prior to prescreening would have greatly increased the risk of fraud, future online studies will likely have to consider the impact of potential selection bias. We recommend that researchers work on methods to enroll and engage actual participants, as once a participant engages, the likelihood of continuing to participate for a year is high.

However, even with this sample size and retention rate, suicide attempts remained a rare outcome. Because of the relative sparsity of suicide attempts among participants, we anticipate that models developed to detect rare outcomes, such as anomaly detection techniques [100], will be most applicable to predicting attempts directly. However, on account of our high survey completion rate, it may also be possible to develop models of suicidal ideation and assess the extent to which predictions of severe ideation anticipate suicide attempts. In addition to sparsity at the outcome level, our data also exhibited sparsity at the feature level—the vast majority of internet searches did not concern content related to suicide. Natural language processing technologies present avenues through which to amplify informative signals by focusing modeling efforts on suicide-related searches [52].

It is well-known that fraudulent respondents and bots are a common problem with remote recruitment [78,80,101-106]. These have the potential to undermine data quality and result in several protocol changes to minimize the costs and burden of these respondents [105,107,108]. In this study, we used recruitment platforms that were noted, at the time, for their intensive screening procedures (eg, Prolific and ResearchMatch) and placed advertisements on advocacy websites (eg, Mental Health America and American Foundation for Suicide Prevention). Of these recruitment platforms, Mental Health America had the lowest percentage of fraudulent actors. To our surprise, ResearchMatch, which is a platform that is funded by the National Institutes of Health and is meant to help consumers find research studies for their condition(s), had the highest percentage of fraudulent actors before we started screening people out for endorsing ResearchMatch as their recruitment source. We are not alone in finding high fraudulent activity from ResearchMatch [107]. While Mental Health America was the most reliable and successful source for recruitment, the platform yielded a sample that skewed young and toward more women vs men or nonbinary people. We also flagged some behaviors as being suspicious (ie, using a virtual private network or inconsistent geolocation) that could have been nonmalicious [102,104]. While some people may have been excluded from the study, not providing this level of security leads to an influx of fraudulent participation [101,102]. Future researchers wishing to conduct remote recruitment may be best served using disease-specific advocacy websites and communities as sample sources. In another entirely remote study, the first author (KAC) recruited over 500 veterans in a year, with minimal fraudulent behavior, through a company focused on social media advertisement [109]. That being said, we are continuing to quantify the impact of fraud, including the monetary cost for fraud mitigation and identifiable differences between fraudulent responders and enrolled participants.

For our work, an important consequence of fraud mitigation is that our focus on an avenue of recruitment with a high yield for genuine participants constrained the demographics of our participant pool. In this context, cross-cutting models trained on the entirety of the pool may generalize poorly to participants from underrepresented demographic groups (eg, men). As the extent to which suicide-related search behavior may differ across these groups is not known, it will be important to compare model behavior across them at the point of validation. Should differences emerge, it may be desirable to develop subgroup-specific models. More broadly, it may be the case that the idiosyncrasies of search behavior favor individualized models over cross-cutting ones.

We recommend that for future research, samples may need to be enriched with certain populations that are less likely to engage in research focused on mental health concerns [110-113]. Open-source methods to detect fraudulent acts need to be developed and continuously updated as fraudulent actors adapt to security protocols [101,102,104,106]. For instance, if commercial tech companies could support scientists in the identification and recruitment of data donors who have legitimate corporate accounts and meet the inclusion criteria for location and available data, this assistance would lower the costs of conducting recruitment.

The inability to easily download Google Search and YouTube data was a challenge (see Multimedia Appendix 1 for an illustration of the download and sharing process). Of those who reached the download screening stage, over three-quarters were unable to download their data, had insufficient data, or did not complete this phase of the study. Over the course of the study, we pursued multiple options to make data donation possible. This included the creation of an app (gTAP) to help participants donate data directly to a secure cloud-based server, providing real-time assistance for donations, and combining data donations with the baseline interview. Google procedure for donation itself proved to be the biggest barrier to data donation. Google Takeout is a complicated tool with high user burden. Additionally, it is not uncommon for Google to change the data structure without warning, which required us to update gTAP several times [114]. Finally, Google made several policy changes about sharing search data during the course of the study such that a period of privacy review (2‐3 weeks per review) was necessary to approve gTAP. We also experienced intermittent failures of Google’s internal data redaction pipeline, which removes identifiable data from search logs. To process data collected during these periods, we developed an internal deidentification pipeline mirroring the redaction elements extracted by Google’s internal pipeline to ensure that future feature extraction methods could be consistently applied. Near the end of the study, we had to transition to a manual backup upload method due to yet another Google policy change. Using this data collection method reduced technical errors that were previously caused by the repeated challenge between Google and gTAP. Future research using Google Search history for disease prediction will need to factor in the many changes Google tends to implement around data donation. Additionally, investigators and funders should be aware of the potential costs associated with app updates beyond the typical costs for operating system updates and should budget for sustained technical support and updates accordingly.

Given these challenges, we recommend that open-source methods to facilitate data donation from search engines be developed to improve retention [115]. Search engine companies could help this process by creating constrained data sharing through standardized, secure APIs such that donors no longer need the technical expertise required to facilitate deposits themselves. Although Google provides a means for each individual to download their own data, the process is onerous for individuals and makes voluntary data sharing complex. Ideally, an individual should be able to provide permission for others to access their Takeout data easily, which is anticipated to both reduce the burden of technical support on research coordinators and improve recruitment rates.

This paper already details the many limitations associated with remote research to collect search engine data with individuals who consent to share their data—including representativeness of the sample and difficulties with obtaining Google data. Additional limitations of the study include the fact that it was conducted in the United States, and therefore participant behavior reported here will be more applicable to US participants. Some of the challenges regarding fraudulent acts reported in this paper may be due to the financial incentive offered for participation. Past studies have found that offering incentives increases the risk of fraudulent acts [101,116,117].

This innovative study demonstrates the potential of fully remote studies to gather a national sample of participants with suicidal ideation beyond health care–based and school-based samples including both survey data and technology use data. We also elucidate the challenges and potential solutions to these challenges—particularly when merging data from technology companies who cannot or will not provide their data to researchers in batch form. Investigators who embark on studies of this nature will need to carefully consider their sources of recruitment, strategies to mitigate fraudulent actors, and the challenges in accessing search engine or other social media data and be realistic about the impact of these enrollment challenges on the generalizability of the sample recruited. We further conclude that providing capabilities for users to access their data is a necessary but inadequate level of support for research on risk detection using logs of internet search behavior. The systems currently in place are difficult to use for many participants and require additional development to facilitate sharing and integration of data for research purposes. Given the potential of these data to inform the prevention of serious events, companies such as Google have a moral imperative to work with scientists to develop algorithms for suicide risk prediction and mechanisms for secure data donation and to share their results with mental health researchers and/or practitioners while ensuring full transparency. Additionally, we call on these companies to work with external ethicists to improve the means by which they share—or do not share—data. Although there have been attempts to engage with these companies to provide external scrutiny of their own suicide risk algorithms [118], to date, these companies have declined. We recommend that scientists and technology companies collaborate in the validation of algorithms using corporate data within a protected enclave [119]. Until such partnerships grow, and technology companies facilitate donation, researchers will need to reserve resources for protection and adequate ongoing engineering support to oversee and correct for changes in search engine data management.

Prediction of proximal risk for suicide is the holy grail of suicide prevention. While we have a limited ability to determine who is at risk for suicide [120], current risk assessment only occurs when the person at risk interacts with school or health systems. Proximal risk for suicide remains elusive, as suicide risk fluctuates over time. We have demonstrated previously that search engine terms and behavior have the potential to both identify those at risk and potentially predict when an event will occur within a month of the event [52]. This ability to identify and intervene could substantially impact suicide rates around the world. To confirm this potential, large-scale research using these data is necessary. In this study, we have demonstrated that large-scale remote research is possible but not without its challenges and costs.