A research team from Carnegie Mellon University and Cleveland Clinic’s Cardiovascular Innovation Research Center leveraged CMR images and cardiologists’ text reports of patients at Cleveland Clinic in Ohio and Florida, United States, and at the University Hospital of Dijon, France, collected between 2008 and 2022.

Trained on more than 11,028 deidentified real patient studies, the researchers’ CMR contrastive language image pretraining (CMR-CLIP) model—a “vision language model” that treats MRI scans as video-like image sequences—learned from over a million continuous sequential images collected during cardiac MRI procedures. Like a cardiologist reviewing a scan, the model captured the heart’s motion and tissue behavior, covering morphology, function, and myocardial viability.

Deborah Kwon, MD, Director of Cardiac MRI at Cleveland Clinic and the study’s clinical lead, said they wanted to automate the pathway from cardiac MRI images to reports, as it’s difficult and resource-intensive to interpret a cardiac MRI study.

“It takes additional training and expertise, and even then there can be some inter-observer variability between different doctors who are reporting them,” she said. “The referring doctor can be confused on next steps for a patient after getting a cardiac MRI. This kind of tool can democratize cardiac scans’ interpretation to community-based hospitals that might not have direct access to expertise.”

By aligning cardiac MRI images with clinical summaries, the model was able to learn from physician descriptions and scan interpretations without needing experts to manually annotate disease-specific abnormalities. It identified several cardiac conditions in a “zero-shot” setting, meaning it had never been directly trained on specific labels but could match images to descriptive prompts like “enlarged left ventricle.”

The model was evaluated on several tasks, including classification of various cardiomyopathies, prediction of ejection fraction, text-to-image retrieval, image-to-image retrieval, CMR image segmentation, and CMR report generation. Left ventricular diseases—the most common usage of cardiac MRIs—were the study’s primary target.

Researchers found that CMR-CLIP can identify common pathologies such as myocardial fibrosis and left ventricular hypertrophy. In real-world clinical tasks, the model achieved impressive accuracies of 88.5% for nonischemic cardiomyopathy, 88.0% for ischemic cardiomyopathy, 96.2% for cardiac amyloidosis, and 98.6% for hypertrophic cardiomyopathy.

David Chen, PhD, who leads the Artificial Intelligence and Informatics in Healthcare Lab at the Cleveland Clinic Diagnostics Institute and is Associate Director of the Cardiovascular Innovation Research Center, emphasized the clinical implications of the work in a press release. “Cardiac MRI interpretation is highly specialized and time intensive. Systems like CMR-CLIP have the potential to support clinicians through automated screening, and interpretation support, particularly in settings where expert readers are limited. Such reader assistant tools are critical to improving patient access to this powerful diagnostic technology.”

With a single example of a condition, the model matched the performance of other systems that typically require dozens of labeled cases. It could also search through large databases of scans using natural language, retrieving similar cases that can potentially help clinicians quickly compare patients with rare or complex conditions. These same capabilities could help support medical training by facilitating clinical exposure to a wider range of conditions.

“This work highlights a new direction for medical AI on how large-scale clinical data can be used to train models without requiring time-consuming manual labeling,” says Kwon. “This technology has the potential to not only improve efficiency but also [the] quality of reporting, to support more consistent and clinically meaningful interpretations, and [to] be an important bidirectional teaching tool in a highly specialized and complex imaging field.”

The research team plans to extend the model to additional cardiac imaging sequences, including myocardial perfusion imaging, T2-weighted imaging, and parametric mapping, and explore applications in automated report generation and interactive clinical decision support systems in resource-limited settings.

“The tool needs to be continually refined. This was all [tested] on retrospective cases. What we want to do next is validate it and start to use it for clinical reporting and see how well the tool performs,” says Kwon.

“What we’re trying to have is a fully automated pipeline where the image acquisition is automated and then the reporting is also automated to help improve the efficiency,” she says. “At the end of the day, a physician still has to go through and review and ensure the quality and the accuracy of the reporting. But if it [the tool] can at least create a rough draft and then the doctor can edit and fine tune it from there, [this] would enhance the efficiency and hopefully improve the quality of the reporting,” says Kwon.