Although Amazon Mechanical Turk facilitates the quick surveying of a large sample from various demographic and socioeconomic backgrounds, it may not be an optimal platform for obtaining reliable diabetes-related information from the online type 1 diabetes population.J Med Internet Res 2023;25:e43593
Patient registries for type 1 diabetes (T1D) are often developed through collaborations among large medical centers . Amazon Mechanical Turk (MTurk), a confidential web-based crowdsourcing platform with more than half a million registered workers [ ], may serve as an alternative route for cost-effectively surveying large samples of patients with T1D receiving care in geographically dispersed health care environments. In this study, we tested the feasibility of using MTurk to gather reliable information from people living with T1D.
In April 2022, we conducted a cross-sectional survey with MTurk workers to evaluate the reliability of their survey responses about T1D using the consistency checks technique . This study received institutional review board approval from the University of Michigan (HUM00212503). A step 1 screening survey was conducted to recruit people with a self-reported diagnosis of diabetes and to assess respondents’ sociodemographic information, health insurance type, and diabetes-related information (ie, type of diabetes, calendar year of diabetes diagnosis, types of health care providers seen for diabetes management, most recent hemoglobin A1c level, and use of insulin and noninsulin diabetes medications). A compensation of US $0.50 was provided for completing this 2- to 3-minute survey. Respondents who reported having T1D in the screening survey were invited to complete the step 2 full survey, which included the same questions asked in the screening survey with additional questions derived from the T1D Exchange core questionnaire [ ]. A compensation of US $3.30 was provided for completing this 15- to 20-minute survey. The workers’ IP addresses and geographical locations were also collected from the MTurk website. Per best practices for MTurk surveying, only workers with a high-quality task performance track record (ie, completing >1000 tasks; >90% of the completed tasks were approved by prior task requesters for payment [ - ]) were allowed to complete the surveys. All questions were set with force response to ensure a 100% response rate. Only US workers were eligible to participate in this study.
Response consistency was determined by comparing responses across the screening and full surveys. Predetermined criteria (ie, matching responses to questions in both surveys about biological sex, education level, insurance type, calendar year of diabetes diagnosis, and current insulin regimen) were set to identify eligible surveys for future research analysis. A descriptive analysis was conducted to calculate the rates of response consistency and eligible surveys.
A total of 1416 respondents completed the screening survey across 4 days. All 508 (36%) respondents who reported having T1D were invited to participate in the full survey, and 229 full surveys (45% of the initial T1D respondents) were completed within 3 days (ie, both surveys were completed within 1 week). After initial quality control, 224 surveys entered the analysis to determine response consistency (). Comparing the screening and full surveys, more than 70% of the responses were identified as having the same MTurk IP address, geographical location, and demographic and socioeconomic information ( ). In contrast, about 20% of respondents consistently reported health insurance or diabetes-related information in both surveys; for example, 26% (n=58) provided consistent responses about the calendar year of diabetes diagnosis. After applying the predetermined criteria for identifying eligible surveys, only about 6% (n=13) of the surveys were determined to be eligible for future research analysis.
Our findings suggest that identifying a population with T1D and gathering reliable information about their disease management through MTurk surveys could be challenging . Despite screening a large number of patients reporting to have T1D, our study was unable to obtain a sufficient sample size of eligible surveys to generate meaningful data. These observations suggest that detailed assessments of patient-reported health conditions and outcomes through MTurk remain limited. However, MTurk could still serve as a strong platform for surveying the online population’s opinions and knowledge [ , ], given the high consistency rates in reporting demographic and socioeconomic information. A potential explanation could be the nature of the recruitment platform: although most MTurk workers may intend to genuinely perform tasks (as demonstrated by the high consistency rate in the sociodemographic information section), they also need to strike a balance between time and completing tasks quickly rather than accurately, as this cohort was recruited to “work” rather than to provide accurate information to make scientific contributions. Thus, if considering using MTurk to survey populations with specific medical conditions, simultaneous conduct of the survey in parallel with other platforms may help to determine the validity of findings from MTurk. Furthermore, prior research has demonstrated discrepancies in patient characteristics between cohorts recruited through MTurk and other platforms [ ], and thus the generalizability of MTurk-based findings also remains to be further evaluated.
In conclusion, MTurk may not be an optimal platform for obtaining reliable responses about diabetes-related information from the online T1D population.
YKL and JP proposed the study and interpreted the results. YKL, SN, and JP designed the study and study instruments. YKL collected study data and conducted the analysis. All authors contributed to the manuscript and reviewed it before submission. The study was supported by the Michigan Center for Clinical and Translational Research Pilot and Feasibility Grant (P30DK092926). YKL was supported by K23DK129724. JP was a VA Health Services Research and Development Research Career Scientist.
We greatly appreciate the T1D Exchange, which provided the full survey questionnaire. We also thank all the study participants, without whom this study would not have been possible.
The data sets generated during and/or analyzed during this study are available from the corresponding author on reasonable request.
Conflicts of Interest
- Beck RW, Tamborlane WV, Bergenstal RM, Miller KM, DuBose SN, Hall CA, T1D Exchange Clinic Network. The T1D Exchange clinic registry. J Clin Endocrinol Metab 2012 Dec;97(12):4383-4389 [CrossRef] [Medline]
- Mortensen K, Hughes TL. Comparing Amazon's Mechanical Turk platform to conventional data collection methods in the health and medical research literature. J Gen Intern Med 2018 Apr;33(4):533-538 [https://europepmc.org/abstract/MED/29302882] [CrossRef] [Medline]
- Schell C, Godinho A, Cunningham JA. Using a consistency check during data collection to identify invalid responding in an online cannabis screening survey. BMC Med Res Methodol 2022 Mar 13;22(1):67 [https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-022-01556-2] [CrossRef] [Medline]
- Chandler J, Shapiro D. Conducting clinical research using crowdsourced convenience samples. Annu Rev Clin Psychol 2016;12:53-81 [CrossRef] [Medline]
- Porter ND, Verdery AM, Gaddis SM. Enhancing big data in the social sciences with crowdsourcing: data augmentation practices, techniques, and opportunities. PLoS One 2020;15(6):e0233154 [https://dx.plos.org/10.1371/journal.pone.0233154] [CrossRef] [Medline]
- Amazon Mechanical Turk. Qualifications and worker task quality. Medium. 2019 Apr 18. URL: https://blog.mturk.com/qualifications-and-worker-task-quality-best-practices-886f1f4e03fc [accessed 2022-03-30]
- DePalma MT, Rizzotti MC, Branneman M. Assessing diabetes-relevant data provided by undergraduate and crowdsourced web-based survey participants for honesty and accuracy. JMIR Diabetes 2017 Jul 12;2(2):e11 [https://diabetes.jmir.org/2017/2/e11/] [CrossRef] [Medline]
- Bardos J, Friedenthal J, Spiegelman J, Williams Z. Cloud based surveys to assess patient perceptions of health care: 1000 respondents in 3 days for US $300. JMIR Res Protoc 2016 Aug 23;5(3):e166 [https://www.researchprotocols.org/2016/3/e166/] [CrossRef] [Medline]
- Harris JK, Mart A, Moreland-Russell S, Caburnay CA. Diabetes topics associated with engagement on Twitter. Prev Chronic Dis 2015;12:E62 [http://www.cdc.gov/pcd/issues/2015/14_0402.htm] [CrossRef] [Medline]
- Freeman-Hildreth Y, Aron D, Cola PA, Wang Y. Coping with diabetes: provider attributes that influence type 2 diabetes adherence. PLoS One 2019;14(4):e0214713 [https://dx.plos.org/10.1371/journal.pone.0214713] [CrossRef] [Medline]
|MTurk: Amazon Mechanical Turk|
|T1D: type 1 diabetes|
Edited by T de Azevedo Cardoso; submitted 05.12.22; peer-reviewed by B Holtz, A Lam; comments to author 27.07.23; revised version received 01.08.23; accepted 02.08.23; published 18.08.23Copyright
©Yu Kuei Lin, Sean Newman, John Piette. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 18.08.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.