Background: Clinicians, researchers, educators, and administrators have grappled with the implications of generative artificial intelligence (AI) in healthcare since the release of the first publicly available large language model (LLM) in late 2022. Efforts have been directed towards evaluating LLMs’ performance in common clinical and scholarly tasks and debating important considerations including patient safety, information security, bias, liability, competencies, and ethicality, but little has been reported on how individuals use AI in their learning and practice.
Methods: A survey was designed to identify areas in which internal medicine residents are using generative AI tools. Additional questions assessed residents’ opinions about the current state and future applications of LLMs in resident education and physician practice. The survey was distributed to internal medicine residents at 5 internal medicine residency programs across the state of North Carolina.
Results: There were 152 surveys with a near even distribution among PGY year. Many respondents identified an existing role for LLMs in both clinical (56%) and non-clinical medicine (72%), although only 26% reported current use. More individuals perceived a future role for LLMs in clinical (71%) and non-clinical (80%) medicine. While no residents had received formal training, 94% expressed interest in learning more about LLMs and 87% felt that residencies maybe or definitely should have a formal curriculum. Residents lacked confidence in AI with 82% feeling neutral or not confident in the accuracy of output. The top three uses of AI in clinical medicine include forming a differential diagnosis, researching treatment options, and creating patient education materials. Uses in non-clinical medicine were more diverse but spanned components of information acquisition, scientific writing, and creation of educational or teaching material. Residents were well aware of concerns and risks with 93% reporting at least one concern and 80% reporting multiple. Amongst those reporting professional use, 45% used LLMs on a monthly basis or more frequently. Respectively, 33% and 39% have encountered inaccuracies in clinical and non-clinical use. Almost all (96%) report verifying the accuracy of LLM output. Two-thirds of those reporting professional use do not disclose the use of LLMs to their patients.
Conclusions: Many residents lack familiarity and confidence in LLMs and express concerns about the risks and limitations in their current state. At the same time, many residents believe that LLMs have a current role in medicine with more anticipating an evolving importance in the future. Importantly, 25% of residents are already integrating LLMs in their work, a signal that we must be engaging in dialogue and discussion with trainees now. It is clear that as the next generation of physicians experiments with LLMs, they desire guidance on safe and effective use. Most residents proceed with caution, validating the output and scrutinizing for errors; however, this may wane as the technology improves. Our study identifies and qualifies the engagement, application, concerns, and reservations of a critical stakeholder in the future of AI in medicine. We believe these results can help healthcare systems, physician organizations, and medical education communities as they contend with how to safely educate, implement, regulate, and monitor the use of LLMs.