Background: A growing body of evidence suggests that selecting graduating medical students for residency interviews based solely on USMLE scores or clinical grades does not adequately capture the breadth of qualities needed to succeed as a resident as outlined by the ACGME competencies. Step scores have been shown to correlate with ITE scores and boards pass rates, but not clinical skill acquisition (McGaghie 2011). In addition, White et al. found a positive association with USMLE scores and socioeconomic privilege (White 2002). Clinical rotation grades as well as the summary word on the medical student performance evaluation (MSPE) have also been recently shown to have serious racial bias (Low 2019). Residency evaluation metrics such as milestones are based on ACGME competencies and have not been correlated with Step score or match list ranking. To shift away from filtering based solely on scores, Prober et al implored for “more standardized modes of assessment and reporting that are readily sortable” to make comparing these skills in large numbers of applicants valuable (Prober 2016).
Purpose: Our project used machine learning and natural language processing (NLP) to build an algorithm that creates a score for each ERAS application based on different values derived from the ACGME competencies, which can be used as a new screening tool for applications. We hypothesize that this values based approach will provide a meaningful way to screen applicants, mitigate bias, and better predict resident success by systematically incorporating holistic core values into the initial stage of the residency selection process.
Description: We developed a novel systematic holistic method to screen residency applications that aligns with the ACGME core competencies as well as the values of our Internal Medicine/Pediatrics (Med/Peds) residency program at Utah. These values included: academic strength, compassion/empathy, communication, work ethic, teamwork, leadership/initiative, self-awareness/humility, justice/diversity/equity/inclusion, professionalism, and flexibility/adaptability. Partnering with the Department of Biomedical Informatics at the University of Utah, we used supervised machine learning to teach a computer algorithm to recognize these values, training the algorithm through 2200 manual annotations of applications from the 2018 ERAS application cycle, looking specifically at the MSPE, letters of recommendation, experiences, and personal statement sections. We validated the machine learning algorithm using a human-scored rubric for each value.
Conclusions: We provide a novel tool that can assess multiple applications quickly using NLP and score them based on values that encompass many crucial domains for a successful Med/Peds physician. The next steps will be expanding this to our categorical programs, where it will be even more valuable given they receive over 1,000 applications each year. It could be used both to select applicants for interviews as well as developing a rank list. We hope it will serve to evaluate applications more holistically, increase diversity, and decrease emphasis on step scores. We plan to share our characteristics with applicants to make the selection process more transparent, and integrate these values into our interview process. Future studies will also look retrospectively at the milestones scores for our residents and compare these to this holistic assessment to determine if these characteristics predict residency success.
