This cross-sectional study used secondary clinical data from electronic health records (EHRs) capturing the Alcohol Symptom Checklist as it was completed during routine care at Kaiser Permanente Washington (KPWA), an integrated health care system in Washington state. KPWA conducts annual behavioral health screening for alcohol, drug use, and depression in primary care [8,9]. Clinical workflows at KPWA for population-based screening and assessment typically occur as follows, although variation in clinical practice is expected: All patients who have not completed screening in the past year are sent a behavioral health screener through the health system’s online patient portal before scheduled primary care appointments as part of electronic check-in. If high-risk drinking is reported during online screening, the Alcohol Symptom Checklist automatically displays in the portal (with exceptions; Multimedia Appendix 1). Patients who do not complete screening online are typically prompted by check-in staff or medical assistants (MAs) to complete screening in clinic on paper during primary care appointments, with results entered into EHRs by MAs. If high-risk drinking is reported in clinic, the EHR prompts MAs to give patients a paper-based version of the Alcohol Symptom Checklist to complete in clinic [9]. Patients complete in-clinic screening and assessment for several reasons: some may not have set up their online patient portal, others may need translated versions in a language other than English, while others choose not to complete the checklist online. During the study period, approximately 90% of all patients with a primary care appointment completed alcohol screening and approximately 80% of primary care patients reporting high-risk drinking completed the follow-up Alcohol Symptom Checklist.

Adult patients (aged ≥18 years) were included in this study if they completed a primary care appointment (phone, video, office) during 2021 and were due for annual screening, reported high-risk drinking on the alcohol screening measure (AUDIT-C score ≥7 [10-13]; Multimedia Appendix 2), and completed the Alcohol Symptom Checklist. Checklists with missing items were excluded (Multimedia Appendix 1). If patients completed ≥1 checklist in clinical care, a single checklist was randomly selected for analysis.

This deidentified data-only study was approved by the KPWA Health Research Institute Institutional Review Board (reference 1481289-1) with waivers of consent and Health Insurance Portability and Accountability Act authorization.

The Alcohol Symptom Checklist is a valid [6] and reliable [7] 11-item questionnaire assessing AUD criteria over a past-year timeframe, mirroring the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5; Multimedia Appendix 3) [14]. Patients self-reported whether they experienced each of 11 DSM-5 AUD criteria (yes/no). Summed scores ranged from 0 to 11, reflecting the number of AUD criteria endorsed and informing the severity of AUD (per the DSM-5, mild=2-3, moderate=4-5, and severe=6-11 criteria) [14].

Modality of administration was coded as online or in clinic (Multimedia Appendix 1).

EHR-documented demographics included age, sex, race, ethnicity, and need for a language interpreter.

Descriptive statistics characterized the demographics of patients completing online and in-clinic checklists. Logistic regression models estimated the prevalence of each AUD criterion and of symptoms consistent with mild, moderate, or severe AUD across modalities, adjusted for demographic characteristics that significantly differed between groups.

Main analyses used item response theory [15] to evaluate differential item functioning (DIF) on checklist items across modalities (Multimedia Appendix 4 [6,14-20]).

DIF can be statistically significant without having a clinically meaningful impact on determination of AUD severity [15]. Clinicians typically make diagnostic and treatment decisions based on total scores rather than individual items because the number of criteria informs whether AUD is likely present (≥2 criteria) and its severity [14]. We evaluated the cumulative impact of DIF by estimating how total scores would be expected to change due to DIF if a patient with the same level of latent AUD completed checklists online vs in clinic (Multimedia Appendix 4 [6,14-20]).

The Alcohol Symptom Checklist administered online performed similarly to the checklist completed in clinic. Small differences in the prevalence of item endorsement (<8%) did not result in differential AUD or AUD severity across modalities. While DIF across modalities was identified for 4 items, it did not meaningfully impact total scores, which are used by clinicians to diagnose AUD. The lack of clinically meaningful impact of DIF across modalities supports the use of online portals or in-clinic paper forms for administering the Alcohol Symptom Checklist in primary care to facilitate accurate AUD diagnosis.

Our findings are consistent with prior work suggesting pragmatic symptom-based checklists have high test-retest reliability [7] and perform similarly across patient demographic characteristics [6,20] and treatment status [21]. Our study adds to the growing body of evidence supporting the validity and reliability of the Alcohol Symptom Checklist completed in real-world routine care settings.

Our findings have important implications for implementing assessment in health care. In-clinic administration may be preferable for clinics that lack online resources (eg, web developers) or online infrastructure (eg, patient portals) to support integrating questionnaires into EHRs and remote workflows. When these resources are available, online administration may be preferable to help streamline workflows, reduce staff burden, and ensure more complete responses (eg, fewer missed items).

Both assessment modalities allow patients to directly report AUD symptoms (rather than staff asking the questions), which decreases stigma and improves screening quality, screening feasibility, and patient comfort [22]. Assessment of AUD symptoms remotely via an online portal may further reduce stigma and increase comfort in self-reporting alcohol-related concerns [23].

The heterogeneity of patients who completed the Alcohol Symptom Checklist online versus in clinic is an important finding that underscores the need for both modalities to avoid inequitable identification of AUD. Young adults (aged 18-24 years), older adults (aged ≥65 years), patients of minoritized racial and ethnic backgrounds, and patients who needed an interpreter were more likely to complete checklists in clinic. As digital tools become increasingly common in health care [24], offering both modalities may help minimize inequities. For example, online portals may be more accessible to patients who do not access in-person services (eg, due to logistical, financial, health status, or other factors), while the in-clinic modality may have greater reach for patients without reliable internet or digital devices, who have lower digital literacy, who need translated materials, who do not use patient portals, and who may be more comfortable responding when they can ask questions or explain responses to a health care provider [25].

This study has limitations. Patients may have underestimated or underreported their alcohol use and AUD symptoms. Racial and ethnic diversity were limited. The sample included patients who completed annual screening at a primary care appointment and thus represent the population that uses care but may not represent other groups that do not receive primary care. Results may not generalize to other populations or health systems. The study has noteworthy strengths. While processes for completing the Alcohol Symptom Checklist online versus in clinic differed, these differences reflect actual clinical assessment procedures, providing high external validity regarding how checklists perform under real-world conditions. The sample size was large and powered to detect small amounts of measurement invariance. Future studies should assess whether modality of AUD assessment differentially impacts diagnosis and treatment.

Remote completion of the Alcohol Symptom Checklist online by primary care patients performed similarly to a valid [6] and reliable [7] paper-based version completed in clinic. Future studies should evaluate methods to support health care providers in following and treating patients with AUD through in-person and virtual care.