Background: Routine patient care including vital signs checks, lab draws, medication administration, during the night contributes to the already disturbed sleep of inpatients. This study aimed to assess the performance of automated risk scores to stratify the risk of an overnight deterioration to better inform letting low-risk patients sleep and more intensively monitoring and/or intervening upon those with higher risk.
Methods: Data was obtained by automated export from the electronic health record (EHR). Inclusion criteria were non-pregnant adults, admitted to the hospital with a level of care of medical/surgical, telemetry, or intermediate intensive care unit (ICU) (e.g., not observation, ED, or ICU). Automated scores of interest included an institutional modification of the Modified Early Warning Score (jMEWS), the Epic Deterioration Index (DI), and the Cardiac Arrest Risk Triage (CART) score (calculated after data export based on requisite data). To assess clinical deterioration, outcomes of interest included new overnight medications, telemetry upgrade, intensive care unit transfer, rapid response team (RRT) activation, code, death, and composites of these outcomes. These outcomes were time-bound between 11 PM and 8 AM (i.e., to determine if an outcome would occur had a patient been allowed to sleep without disruption). Events that occurred outside of this window, including RRT and code outcomes or deaths, were excluded. Prior to full analysis, to ensure the accuracy and validity of the data extracted through the algorithm, a researcher conducted a manual validation process to confirm its reliability and 37 of 37 manual verifications suggested the automated data was reliable within its constraints.
Results: From 2/28/22-5/15/23 across six hospitals, 474,225 patient-days were assessed. Upon filtering to “Inpatient” patient class and non-ICU floors, 359,409 patient-days remained. Data missingness was 135/359,409 (0.04%), 1,165/359,409 (0.3%), and 33,458/349,409 (9%) for jMEWS, DI and CART, respectively. For the most restrictive composite outcome of interest — code or death, the scores’ area under the curve (AUC) were DI (0.93), jMEWS (0.89), CART (0.86). For the most inclusive composite of interest (any outcome except new meds) the AUCs were DI (0.74), jMEWS (0.73), CART (0.74). The percentage of total patients and rate of code or death/1,000 patient-nights for bins of the DI are: ≤20 (20.7% of all patients): 0.00, >20-≤25 (27.3%): 0.06, >25-≤30 (17.4%): 0.16, >30-≤35 (11.1%): 0.33, >35-≤40 (7.9%): 0.53, >40-≤45 (4.9%): 1.20, and >45 (10.8%): 5.28.
Conclusions: While all three scores performed well on the most restrictive compositive outcome of interest – death or code, it is notable that the DI performed the best (AUC = 0.93). Its ubiquity of integration into an EHR used at hundreds of hospitals and high discriminatory capabilities make it useful for national and international efforts to better identify patients who are at low risk for deterioration overnight and may be appropriate to prioritize a less disruptive night of sleep; for those at higher risk, they may need further diagnostics, interventions, and/or monitoring. For instance, patients with a DI score ≤25 account for 48% of all patients and have a risk that is 152x lower for an overnight death/code than patients with a DI score >45 (11% of all patients). Hospital systems can create clinical decisions support systems that balance local risk/reward tolerance to set thresholds to help clinicians to better risk stratify overnight.