Background: Residents receive little feedback on their clinical reasoning (CR) documentation due to time constraints of supervisors and lack of a shared mental model. We developed an innovative workplace-based assessment tool using machine learning (ML) to provide feedback on CR documentation. Here, we describe the impact of feedback using this assessment tool.
Methods: In earlier phases, we developed a ML model that categorizes medicine resident admission notes as documenting high-quality vs. low-quality CR. All H&Ps written by residents are automatically analyzed by the ML model and displayed on a dashboard accessed via EPIC. In this study, we assessed the impact of implementation of feedback via this dashboard on residents’ CR documentation. The majority of H&Ps are written on night shifts which occur in two week blocks. Half-way through the two week block, the residents received an introductory email including a brief powerpoint overviewing the importance of high-quality documentation, a shared mental model for high-quality CR documentation, and an overview of the dashboard. Residents then had the opportunity to review their dashboard data and complete a post-survey documenting their action plans for improvement and provide feedback on the dashboard.
Results: 27 residents received the intervention from 4/2021-6/2021, 19 of which accessed the dashboard. 7 of those residents were excluded from data analysis as they did not have both pre- and post-intervention data. There was not a significant difference pre- compared to post-intervention in CR note quality. 46% (SD 26%) notes were high-quality pre- compared to 44% (SD 25%) post-intervention. However, there was a significant correlation between number of dashboard views and improvement in note quality with a pearsons correlation coefficient of 0.70. 7 residents completed the post-survey documenting specific action plans for improvement: “I will try to always include at least one alternative diagnosis with at least two data points explaining my choice.” Feedback for improvement on the dashboard included the desire for more specific feedback and suggestions to use this tool with faculty to review notes.
Conclusions: While we did not see an improvement in note writing quality, there was a trend in improvement post-intervention in those residents who accessed the dashboard more than once. Adding this feedback modality to a program of assessment can lead to increased frequency of feedback and ideally improve documentation quality. Next steps are to continue to collect data on impact, to iterate on the model to provide more specific feedback, and to train faculty to use this tool with residents.