Unveiling GPT-4V's hidden challenges behind high accuracy on USMLE questions: Observational Study

Zhichao Yang   ^{1 *} , PhD ;   Zonghai Yao   ^{1 *} , MS ;   Mahbuba Tasmin   ¹ , MS ;   Parth Vashisht   ¹ , MS ;   Won Seok Jang   ² , RN, MS ;   Feiyun Ouyang   ² , PhD ;   Beining Wang   ³ , BS ;   David McManus   ⁴ , MD ;   Dan Berlowitz   ^{5, 6} , MD, MPH ;   Hong Yu   ^{1, 2, 6, 7} , PhD

¹ College of Information and Computer Science, University of Massachusetts Amherst, Amherst, MA, United States

² Miner School of Computer & Information Sciences, University of Massachusetts Lowell, Lowell, MA, United States

³ Shanghai Medical College, Fudan University, Shanghai, China

⁴ Department of Medicine, University of Massachusetts Chan Medical School, Worcester, MA, United States

⁵ Department of Public Health, University of Massachusetts Lowell, Lowell, MA, United States

⁶ Center for Biomedical and Health Research in Data Sciences, University of Massachusetts Lowell, Lowell, MA, United States

⁷ Center for Healthcare Organization and Implementation Research, VA Bedford Health Care System, Bedford, MA, United States

*these authors contributed equally