Despite excitement around artificial intelligence (AI)–based tools in health care, there is work to be done before they can be safely deployed. Addressing dataset diversity and racism perpetuated by large language models is paramount [1-3]. The absence of diverse patient voices in AI discussions is a pressing matter, and current studies have been limited in diversity [4,5]. Our study examined the perspectives of racial and ethnic minority patients on the use of health data in AI.

Overall, 230 participants enrolled, and 199 surveys were completed (Table 1). While familiarity with AI was high, a smaller proportion of participants understood how AI can be used in health care (152/199, 76.4%), and an even smaller proportion understood how AI can be applied to dermatology (133/199, 66.8%). These outcomes did not vary significantly when stratifying by demographics such as race, age, gender, income, insurance type, or schooling (all P>.05).

Most participants (139/199, 69.8%) agreed that they trusted the health care system to treat their medical information with respect, which varied significantly by income (P=.045). Patients with lower incomes often experience more structural barriers to health care, and this could be one factor contributing to our finding that they are not as likely to trust the health care system. Only 64.3% (128/199) participants felt comfortable with their medical information being used to build AI tools. Most (181/199, 91%) want to be notified if their medical information is used in such a way. This varied by race (P=.002) and age (P=.03). Most (166/199, 83.4%) agreed they should be compensated if their medical information is used to develop AI, which varied by age (P=.045; Figure 1).

To our knowledge, this is the first study focused on understanding opinions about health data use for AI among people who identify as Black, Latinx, Indigenous, or Asian. Similar studies have had limited diversity [4,5]. It is important to capture the opinions of diverse groups because the inclusion of their data is essential for building equitable AI tools; however, historical harms have made robust inclusion challenging. For racial and ethnic minority communities, historical experiences of racism, discrimination, and exploitation (eg, medical experimentation on populations without informed consent) may contribute to distrust in the health care system and, hence, decrease their comfort level with their data being used to develop AI systems. This suggests that AI development may not be inclusive, ethical, or aligned with the concerns of racial and ethnic minority communities enough to be safely deployed in health care settings. Ignoring these concerns could deepen distrust in AI, leading to lower rates of research participation in these already underrepresented communities, deepening biases and inequities in industries where AI is being deployed.

The educational gaps in AI uncovered by this inquiry should be addressed, as a fundamental understanding of AI is essential for patients to offer fully informed consent for health data use in AI development. There are existing initiatives addressing these educational gaps [6].

Most participants agreed that they should be notified if their data are used to build AI tools, which is not the current convention. For example, in dermatology, photos taken for clinical monitoring can be repurposed as data in AI tools without patient permission as long as they are “deidentified.”

Over 80% of participants agreed that they should be compensated when their data are used to build AI tools. Community-based participatory research shares findings and benefits with study participants. Sharing revenue with patients who contribute their data to build these valuable tools is thematically similar to sharing benefits of a work product and is something that should be explored.

Limitations include the study population. Participants were recruited from an academic medical center and research database, meaning that they may not represent the broader population. Future research should include a more diverse and representative sample from various backgrounds and regions to enhance generalizability.