Using a Diverse Test Suite to Assess Large Language Models on Fast Health Care Interoperability Resources Knowledge: Comparative Analysis

Ahmad Idrissi-Yaghir   ^{1, 2 *} , MSc ;   Kamyar Arzideh   ^{2, 3 *} , MSc ;   Henning Schäfer   ^{2, 4} , MSc ;   Bahadir Eryilmaz   ² , MSc ;   Mikel Bahn   ² , MSc ;   Yutong Wen   ² , MSc ;   Katarzyna Borys   ^{1, 2} , MSc ;   Eva Hartmann   ⁵ , MSc ;   Cynthia Schmidt   ^{2, 4} , MD ;   Obioma Pelka   ^{1, 2} , PhD ;   Johannes Haubold   ^{1, 2} , MD ;   Christoph M Friedrich   ^{6, 7} , PhD ;   Felix Nensa   ^{1, 2} , MD ;   René Hosch   ^{1, 2 *} , PhD

¹ Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany

² Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany

³ Department of Central IT, Data Integration Center, University Hospital Essen, Essen, Germany

⁴ Institute for Transfusion Medicine, University Hospital Essen, Essen, Germany

⁵ Department of Pulmonary Medicine, University Hospital Essen - Ruhrlandklinik, Essen, Germany

⁶ Department of Computer Science, University of Applied Sciences and Arts Dortmund, Dortmund, Germany

⁷ Institute for Medical Informatics, Biometry, and Epidemiology, University Hospital Essen, Essen, Germany

*these authors contributed equally