The Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for the Evaluation of Medical Large Language Model Question-Answer Studies: Development and Pilot Validation

Carrie Ye   ^{1, 2, 3} , MPH, MD ;   Joseph Ross Mitchell   ^{1, 4} , PhD ;   Daniel C Baumgart   ¹ , MBA, MD, PhD ;   Zechen Ma   ¹ , MD ;   Angela Lim Fung   ^{1, 5} , MSc ;   Daniela Garcia Orellana   ¹ , MSc ;   Juel Chowdhury   ^{1, 6} , MBBS, MPH ;   Abdullah Abass   ¹ , MD ;   Steven Katz   ^{1, 7} , MD ;   Jacob L Jaremko   ¹ , MD, PhD ;   Pierre Boulanger   ¹ , PEng, PhD ;   Claire E H Barber   ⁸ , MD, PhD ;   Gillian Lemermeyer   ¹ , RN, PhD ;   Hosna Jabbari   ¹ , PhD ;   Lili Mou   ^{1, 4} , PhD ;   Maryam Mirzaei   ⁹ , PhD ;   Mary Waithera Beckett Githumbi   ⁵ , MBA ;   Puneeta Tandon   ¹ , MSc, MD ;   Randy Goebel   ^{1, 4} , PhD ;   Rhys Clark   ³ , BSc ;   Whitney Hung   ³ , MD ;   Marjan Abbasi   ¹ , MD ;   Farhad Maleki   ⁸ , PhD ;   Scott Klarenbach   ¹ , MSc, MD ;   Mohamed Abdalla   ^{1, 4} , PhD

¹ University of Alberta, Edmonton, AB, Canada

² Arthritis Research Canada, Vancouver, BC, Canada

³ Alberta Health Services, Edmonton, AB, Canada

⁴ Alberta Machine Intelligence Institute, Edmonton, AB, Canada

⁵ University of Toronto, Toronto, ON, Canada

⁶ University of Oxford, Oxford, United Kingdom

⁷ Queen's University, Kingston, ON, Canada

⁸ University of Calgary, Calgary, AB, Canada

⁹ NAIT Applied Research, Edmonton, AB, Canada