ACGME Milestones and Entrustable Professional Activities (EPA) have emerged as benchmarks for the professional development of house staff. Residents must demonstrate expertise in these patient care activities in order to progress in training, and thus, programs will require tools for direct observation in order to evaluate these skills. These tools must evaluate milestone‐specific performance as well as provide timely summative and formative feedback. We report the use of 1 such tool, the UPDATED sign‐out audit tool, to evaluate ACGME milestones related to handoffs.
This study occurred in the internal medicine residency program at the University of Chicago, an academic medical center whose electronic health record features an electronic sign‐out. The UPDATED audit tool was used to evaluate intern electronic sign‐outs on inpatient internal medicine services. This tool evaluates several key ACGME milestones, including accurate documentation (ICS‐F1, P‐A1), appropriate synthesis and definition of clinical problems (PC‐C1), and effective communication during transitions of care (ICS‐C1). All 45 interns were assigned independent review of 2 electronic sign‐outs from July to October by 1 of 8 faculty auditors (4 chief residents and 4 hospitalists). Seven specific sign‐out attributes were scored, and an overall score of 0–9 was assigned. Audits scoring 0–3 were categorized as “poor,” 4–6 as “fair,” and 7–9 as “good.” Ratings among faculty members were compared for consensus.
Of the 45 interns, 32 had complete audits, 6 had partial audits, and 7 were unable to be audited during the initial period. Interns were identified for immediate feedback if they received 2 “fair” scores or 1 “poor” score. If sign‐outs were unable to be fully audited or received at least 1 “fair” score, interns were assigned additional review in subsequent months. Sign‐outs were considered satisfactory if both audits were scored as “good.” Of the 32 completed audits, 16% of interns (n = 5) were identified for immediate feedback, and 41% (n = 19) for additional future review. The remaining interns (44%) were satisfactory. Audits were incorporated into intern semiannual performance reviews within the house staff evaluation committee. Interns achieving a satisfactory audit were designated as “competent,” those awaiting further review as “undergoing further evaluation of competence,” and those flagged for immediate feedback as “received immediate feedback and undergoing further evaluation of competence.” The 13 interns not fully audited were designated as “waiting to be evaluated.” A second round of audits will be done from December to April.
The UDPATED sign‐out audit tool is a feasible and effective way to use direct observation to address resident achievement of competency in ACGME milestones. The UPDATED tool allows incorporation of milestones in house staff evaluation and is well suited to serve as an assessment for the proposed EPA encompassing patient handoffs.