Background: Adverse events (AEs) following hospital discharge occur in 19–28% of patients and pose significant challenges, particularly among those with multiple chronic conditions (MCCs). Accurate prediction of these events is essential for the growing population of adults with MCCs. Conventional risk models that rely on structured electronic health record (EHR) data often fail to account for cognitive, physical, and psychosocial factors, which are more effectively identified within unstructured documentation. As part of an AHRQ-funded investigation, we evaluated whether a large language model (LLM) could extract relevant variables from clinical narratives to predict post-discharge AEs.

Methods: This secondary analysis included 293 adults with at least two chronic conditions who were discharged from general medicine services at an academic medical center in Boston, MA, between February 2022 and August 2023. An LLM (GPT 4o, Microsoft Azure) was prompted to extract information on cognitive, physical, and psychosocial impairments in structured format (e.g., concern for cognitive status present, normal function, uncertain) from physical therapy (PT), social work (SW), and discharge notes for all cases (Figure). Human-LLM agreement was independently assessed by two clinicians using notes from a random sample of five cases, reported as percent agreement. All cases underwent a validated process where two clinicians independently confirmed the occurrence of AEs within 14- or 30-days post-discharge by chart review using the EHR (Epic Systems, Inc.). For each case, the presence or absence of LLM-detected impairments across the three types were dichotomized and summed as a predictor variable with integer values ranging from 0 to 3. Logistic regression was used to model AE outcomes, with summed predictors analyzed both as nominal and ordinal categorical variables to assess trend. Statistical analyses were performed using R, with significance at p < 0.05.

Results: Among 293 patients: mean age was 56.7 (SD 15.9); 179 (61.1%) were female; 191 (65.2%) were White, Non-Hispanic; 288 (98.3%) spoke English; 159 (54.3%) had private insurance; 137 (46.8%) had annual income below $98.3K; 147 (50.2%) had three or more chronic conditions; 103 (35.2%) had a PT note; 87 (29.7%) had a SW note; all had a discharge summary; and the mean readmission risk score was 23.1 (SD 13.8). LLM-detected cognitive, psychosocial, and physical impairments were observed in one or more notes from 102 (34.8%), 178 (60.8%), and 210 (71.7%) patients, respectively. Clinician raters agreed with LLM-detected impairments in 37/42 (88%) notes. Within 14- and 30-days post-discharge, 66 (22.5%) and 90 (30.7%) patients experienced an AE, respectively. Presence of all three predictors was significantly associated with increased odds of 30-day AEs (OR 2.79 [1.07–7.27], p=0.04). Trend analysis approached statistical significance for 14-day AEs (OR 1.05 [1.00–1.10], p=0.06) and achieved significance for 30-day AEs (OR 1.06 [1.01–1.12], p=0.04) for each one-point increase in the summed predictor.

Conclusions: Among hospitalized adults with MCCs, cognitive, psychosocial, or physical impairments were frequently identified in unstructured clinical documentation. LLM-detected impairments were associated with increased odds of 30-day post-discharge AEs, with a trend towards increased risk with the total number of impairments. LLM-based methods offer potential for supporting scalable efforts to improve prediction of AEs during care transitions.

IMAGE 1: FIGURE. Example of LLM-prompt, unstructured clinical narrative, and LLM output

IMAGE 2: Table. LLM-detected cognitive, psychosocial, and physical impairments based on physical therapy, social work, and discharge notes and odds of 14- and 30-day post-discharge adverse events (AE)