Background: Standard risk prediction tools including the quick-Sequential Organ Failure Assessment (qSOFA) have been validated using in-hospital outcomes (mortality [Died-Hosp] or intensive care unit stay >72 hours [ICU72]). Yet, risk models are often applied to predict outcomes beyond hospitalization, and it is unclear what effect outcome selection has on model performance. We derived independent robust risk models to predict Died-Hosp, composite of death or ICU72 (Died-Hosp/ICU72), 30-day (Died30) and 90-day (Died90) mortality outcomes and compared performance when each model’s coefficients were applied to predict more distal outcomes. We hypothesized model performance would decrease significantly when applied at 90 days.
Methods: This was a retrospective cohort study of adults hospitalized with suspected serious bacterial infection (first antibiotics ordered and cultures drawn <24 hours from admission) between January 2014 and September 2017 at 12 acute care hospitals. We developed four independent logistic regression models for Died-Hosp, Died-Hosp/ICU72, Died30, and Died90 (variables included demography, comorbidities, prior healthcare utilization, and labs, vitals, and clinical orders <24 hours from admission) and qSOFA alone (≥2 of hypotension, tachypnea, altered mental status) for predicting the event of interest. All data were extracted from the healthcare system’s enterprise data warehouse, while 30- and 90-day outcomes were captured via linked national death records. We applied model coefficients across different outcomes and compared area under the receiver-operating-characteristic curve (AUC) using DeLong’s test.
Results: Among 52184 admissions, 2030 (4%) Died-Hosp, 5006 (10%) died Hosp/ICU72, 3417 (7%) Died30 and 5655 (11%) Died90. Median age was 60 years, 9829 (19%) had qSOFA≥2, and 25262 (48%) had a Charlson comorbidity score ≥5 (Table1). AUCs of qSOFA, Died-Hosp, Died-Hosp/ICU72, Died30, and Died90 models were 0.71, 0.91, 0.91, 0.89, and 0.87, respectively. Model performance decreased significantly when applied at each more distal outcome (p<0.01). We observed the largest decline in performance when applying Hosp/ICU72 model coefficients to predict 90-day mortality (-0.13, p<0.01).
Conclusions: Our study quantifies the performance trade-off when applying models developed using in-hospital mortality to predict more distal outcomes. A robust risk model featuring acute and chronic factors trained to predict in-hospital mortality can also predict 90-day mortality well. Yet, optimal 90-day mortality prediction requires model derivation based on the event of interest.
.png)