We convened a Delphi panel to understand the diverse experts’ views about the ethical considerations arising with a clinical trial of an autonomous AI that screened for DR in pediatric patients. The Delphi methodology was used because it is an established, structured, and iterative process that involves a panel of relevant experts to identify areas of consensus and dissent regarding an issue [22-24]. We used a modified Delphi approach, conducted in 3 rounds (Figure 1): a first-round survey; a second-round survey incorporating summary results from the first round; and a final virtual meeting round in which the expert panel reviewed consensus statements and further developed recommendations. See Checklist 1 for the DELPHISTAR reporting guidelines checklist.

Delphi panels can range in size from 10 participants to larger groups. We aimed for a panel of 12 to 15 participants. Delphi studies in the health sciences involve an average of 15 participants [25,26]. This number is thought to balance the needed range of expertise and a panel size suitable for engaged conversation for the final round meeting [24]. We identified potential Delphi participants by searching the biomedical literature in the PubMed database to identify experts in areas relevant to questions of AI use in retinopathy, including in AI and data science, ophthalmology, public health policy, law, bioethics, and patient advocacy (Table 1). Several potential participants were selected for each category of expertise and ranked based on factors such as years of experience, previous work on similar projects, and publications. Email invitations were sent based on the rankings to fill each relevant area of expertise. Twenty-four invitations were sent out in total. Ten potential participants did not respond or declined, citing time constraints. Fourteen experts agreed to participate and represented the necessary categories of expertise. Several participants had overlapping relevant areas of expertise. For example, 2 of the 3 individuals with bioethics expertise also represented expertise in AI and clinical ophthalmology. Participants were given information regarding the Delphi study and consented to participate in the study. Key considerations when selecting panel size include ensuring a diverse range of expert perspectives, timing in administration of iterative rounds of surveys, and coordination of scheduling a final group discussion with multiple experts.

The protocol for this study was approved by Stanford’s Institutional Review Board (Protocol 69318). Informed consent was obtained, with a waiver of documentation approved by the institutional review board. Data from surveys 1 and 2 were anonymized. No compensation was used.

Of the 14 experts who agreed to participate in the panel, 12 (85.7%) responded to survey 1. The questions and exemplar quotes from survey 1 respondents are presented in Table 2.

Participant responses regarding the ethical principle of “Clinical and Social Value” indicated several key tensions in applying this principle to the evaluation of an autonomous AI tool. For example, several participants identified the need for ensuring clinical follow-up as a potential challenge to substantiating the clinical value of an AI tool. Most participants broadly saw potential social value, that is, that research should contribute to scientific understanding of health or improve ways of preventing, treating, or caring for people with a given disease to justify participants’ exposure to the risk and burden of research. The social value was reported as the potential to provide a more timely and accurate diagnosis for a disease with treatment available for mitigating further disease progression and to narrow the gap in access to screening for underserved populations. Participants also provided several suggestions for how to evaluate or address social value further, which included community focus groups and a data registry for participating sites.

Most participants raised cost considerations as potentially conflicting with the clinical or social value of the AI tool. At the same time, most participants felt that the cost-benefit analysis of autonomous AI is no different than the assessment of non-AI health care interventions. However, some responses reported that a concern specific to AI tools in health care was the potential impact of patient, public, and providers’ trust in an AI tool on the clinical impact of the tool.

Regarding the principle of “Favorable Risk-Benefit Ratio,” the risk-benefit considerations of autonomous AI were seen to be multifactorial with variable unknowns regarding the introduction of AI into workflows and barriers to data extraction and collection, making risk-benefit analysis difficult. Participants also broadly reported that the consideration of clinical equipoise for AI-based clinical trials was similar to those of non-AI clinical trials.

The issue of health disparities was seen in survey responses regarding several of the principles for ethical research, such as “Clinical and Social Value,” “Fair Subject Selection,” and “Respect for Enrolled Subjects.” The majority of participants had concerns regarding the impact of existing disparities in access to care on the AI tool application development and bias. Participant responses varied on the relevance of access to diagnostic screening without sufficiently addressing barriers to access to therapeutic care. Most participants emphasized that this was not a new issue in clinical research. A few participants posited autonomous AI clinical trials as potential research waste, where resources could be allocated elsewhere to improve care access. Some participants saw access to screening as beneficial, with one stating that screened patients would still have the knowledge of the presence of a health irregularity and may access care down the road.

Of the 14 experts who agreed to participate in the panel, 10 (71.4%) responded to survey 2. Two experts who missed the survey deadline emailed written comments on the survey questions that were incorporated into the survey 2 findings. Textbox 2 summarizes statements of consensus (at least 80% rating agreement), moderate agreement (60%‐80% rating agreement), and divisive statements (with no more than one neutral rating and otherwise panelists evenly split across agree and disagree ratings).

The final round meeting had both synchronous and asynchronous participation by 13 (92.9%) of 14 participants. Panelists identified reducing bias, improving access to care, ensuring downstream access to recommended treatments, promoting transparency regarding the representativeness of the training data, determining prespecified outcomes with respect to the reference standard, having clear pretrial discussion of cost or cost savings, and engaging relevant stakeholders (eg, clinicians and patients) as central priorities. A broader theme derived from the discussion was a tension that although many ethical concerns in AI trials resemble longstanding issues in clinical research, autonomous AI trials may provide an opportunity to address these issues more systematically. The following subsections summarize the combined results.

Participating panelists emphasized that AI tools are distinct from non-AI medical interventions, yet the issues faced in conducting clinical trials are similar. Panelists who expressed the position that AI tools did not present significant differences for clinical trials tended to point out that priority ethical issues for AI, such as bias or informed consent, have also, historically, presented problems for traditional medical interventions. Given the current investment in AI devices, panelists saw there being a potentially useful moment for AI developers to reflect on bias and avoid the development of medical devices that would unintentionally amplify those inequalities. Some panelists pointed toward ways that they saw AI presenting distinct ethical challenges, including clinicians’ limited understanding of potential AI tool limitations and the need to ensure context-specific usability within diverse clinical workflows. Panelists agreed on the existence, but not the magnitude, of these differences and on whether such differences warrant adjustments to the ethical framework for clinical trials. Regardless of the distinctiveness of AI evaluation, expert panelists agreed that efforts to improve clinical trials of AI could also provide an important opportunity to address longstanding ethical issues of bias, informed consent, and access to care.

The expert panel highlighted the need for trials to prespecify outcomes with respect to the reference standard and to describe any challenges in comparison, such as situations in which the current professional medical examination screens for multiple conditions simultaneously, while the AI tool screens for only one condition. Panelists frequently used the term “trial design” broadly to include aspects such as randomization strategy, comparator arm selection, and primary endpoint definitions.

Several of the panelists raised questions of how the autonomous AI tool for DR screening should be compared to screening by an ophthalmologist. Panelists were interested in this comparison as part of considering the broader issue of whether reliance on AI for DR screening as a cost-effective tool in low-resource settings could create or reinforce a lower tier of care. In the vignette considered by the panel, the AI tool had been evaluated against a level 1 reference standard based on clinical outcome, indicating that the AI for DR screening tool was more accurate than a human clinician at screening for DR, as this was raised by one of the panelists. Nevertheless, panelists agreed that trial reports should clearly explain the chosen reference standard and how it relates to the clinical care patients would otherwise receive.

The panelists endorsed several approaches that are meant to address the use of AI across different contexts and for populations that may not be sufficiently represented in training datasets. The clinical trial of an AI tool for DR screening demonstrated that, when placed at point-of-care sites, the tool could improve screening rates among populations historically lacking access, and its performance did not differ across minoritized subgroup populations. However, panelists raised concerns more broadly that a lack of sufficient representation of minoritized populations in training data can lead to systematic bias that undermines the potential benefits of increased screening. There was general agreement among the panelists of the importance of transparency and sharing of information regarding the demographics of training datasets.

The impact of context and integration into workflow on the effectiveness and accuracy of AI tools was highlighted as a particularly challenging issue for evaluation. Panelists supported practices of stakeholder engagement for the development and implementation of AI tools in health care. Panelists noted, for example, engaging clinicians regarding practical aspects of how the AI tool might fit into their workflow could also identify the way that use of the tool in those different contexts could support or create challenges for effective use. One concern raised was that a tool that only evaluates for one condition may not be as useful as a tool that can evaluate several potential eye conditions.

Several panelists discussed how community and/or patient engagement could also be used to help identify factors that impact the usefulness of AI tools for specific purposes and that these kinds of engagement could help identify systemic factors that affect bias and fairness. Panelists agreed that the specific stakeholders needing to be engaged might vary according to the type of tool and purpose. Panelists noted that stakeholder engagement has become more common in recent years for clinical research, and they recognized the need to include such engagement with AI to support these kinds of efforts regularly as part of the process of development and implementation.

Several panelists expressed concern that, generally speaking, AI could make it easier to implement different tiers of care, where the price point of an AI tool could be used to lower costs as well as quality of care for lower-resourced patients. While that was not as likely the case with the specific AI tool for DR screening, given studies found that the tool performed better than clinicians on average, panelists wanted to note this issue for the field in general. At the same time, panelists acknowledged that such concerns regarding cost are not necessarily unique to AI tools within health care. Several panelists suggested that cost savings with an AI tool could present an opportunity within a health care institution or system in ways that benefited underserved patients, such as increasing support for additional access to treatment.

Panelists gave substantial attention to the downstream implications of the AI tool for DR screening and whether evaluation of the tool should account for access to treatment options after a positive screen. Several panelists expressed ethical concerns over an AI tool that might increase the number of people identified as potentially having DR but then leave those people without a way to address their condition. Participants noted that this concern of a diagnosis without access to treatment is certainly not limited to AI tools and that clinical trials are generally not expected to resolve system-level gaps in access. However, the majority of panelists agreed that such downstream implications should be considered when evaluating when and how to use an autonomous AI tool. A few panelists also raised a potential benefit that having an AI screening tool implemented might free up clinician time for treating rather than screening. They noted, however, that whether this benefit would be realized would still depend upon the resources and structure of the health system in which the screening takes place.

The Delphi panel concluded that to apply the 7 ethical principles of research to autonomous AI clinical trials, it is particularly important to incorporate evaluation of potential downstream implications, such as access. Panelists acknowledged that a number of the concerns that they prioritized for ethical clinical research with AI could be applied to conventional medical interventions as well, such as addressing the risk of bias and incorporating engagement with patients and other stakeholders. At the same time, however, the expert panelists overall agreed that addressing the framework for research ethics of autonomous AI clinical trials could also be seen as a greater opportunity to advocate for structural changes that promoted equity more broadly in clinical research and health care.

This Delphi study identified several areas of expert consensus regarding how the 7 NIH ethical research principles should be applied to clinical trials of autonomous AI. Consensus centered on practical features of trial design and reporting, including documentation of training and validation data, prespecified outcomes and reference standards, assessment of performance across populations and settings, engagement with relevant stakeholders, and consideration of cost, access, and follow-up care. Although panelists differed on the extent to which they viewed autonomous AI as distinct from conventional medical interventions, they agreed that many concerns raised by autonomous AI tools reflected longstanding challenges in clinical research. At the same time, panelists viewed the current development of autonomous AI tools as an opportunity to address such concerns more systematically within clinical trial design and evaluation. Overall, these findings suggest that applying established research ethics principles to autonomous AI clinical trials requires evaluating tool performance, while also considering downstream effects on patients, clinicians, and health systems.

These findings extend existing applications of the NIH research ethics principles by specifying how clinical value, scientific validity, and equity should be considered in autonomous AI clinical trials. The panel’s emphasis on downstream implications aligns with prior work emphasizing that ethical evaluation of medical AI depends not only on model performance but also on implementation context and real-world deployment [3,7,29]. This focus is also consistent with qualitative evidence showing that clinical AI implementation is shaped by interdependent factors across multiple stakeholder groups [30] and that developers can recognize potential AI-related harms to patients, groups, and health systems though they vary in their level of perceived responsibility in mitigating harms [31]. The panel’s prioritization of bias and representativeness is further supported by evidence that addressing AI bias requires both technical and social approaches [29,32], that most FDA-evaluated AI tools do not report demographic characteristics of training data [33], and that most AI clinical trials internationally are single-center studies [34]. Qualitative work on AI health datasets also supports the importance of documenting data representativeness and intended use in terms of societal impact [35]. Taken together, our Delphi study and the existing literature support incorporating attention to representativeness, subgroup performance, downstream access, and lifecycle costs into the ethical evaluation of autonomous AI clinical trials, particularly for tools intended for low-resource settings.

Our earlier qualitative study of clinical trials involving AI for DR screening identified a range of ethical challenges that were not fully addressed by existing research ethics frameworks, including issues related to social value, scientific validity, and informed consent [21]. However, the participants in that study were restricted to investigators involved in AI clinical trials for the specific purpose of screening for DR. By engaging a broader, multidisciplinary panel of experts, including experts in health policy and patient advocacy, panelists were able to reach consensus on several key overarching issues affecting the application of ethical research principles to AI clinical trials. The study resulted in the development of 9 key actionable recommendations (Table 3). Implementation of these actions would help regulators and investigators promote equitable potential benefits and reduce potential unintended consequences of autonomous AI clinical trials.

The study has several limitations. While the number of experts on the Delphi panel was within the recommended range for Delphi practices and was highly multidisciplinary, the panel may not represent the full range of perspectives. In addition, all expert panelists were based in the United States, which could limit the international applicability of the recommendations in terms of familiarity with other practice and regulatory contexts. As a Delphi study, the approach is based on the idea that structured discussion and review of the research questions by relevant experts yields practical guidance. Additionally, the regulatory and technological landscape for autonomous AI is rapidly evolving, and therefore, these recommendations may require ongoing refinement as new tools emerge. This study also did not directly address environmental sustainability associated with AI development and monitoring, which could be an area of future consideration.

Autonomous AI clinical trials raise ethical considerations that extend beyond technical accuracy, particularly when tools are intended for use across diverse populations and health care settings. The recommendations developed through this Delphi process suggest that ethical evaluation of these trials should account for the provenance and representativeness of training and validation data, the possibility of biased performance, stakeholder perspectives, access to follow-up care, and cost-related effects. As autonomous AI tools continue to be tested in clinical research, these considerations can help shape regulatory and policy expectations for the ethical development and evaluation of autonomous AI clinical trials.