Background: Hospitalists operate in an ever more complex ecosystem of local policies, procedures, specialty services, facilities and governmental regulations. This knowledge base that a hospitalist needs to function effectively can include both informal know-how and formalized documentation that is spread across myriad resources (1). In many institutions, there is a gap in providing this know-how, as traditional medical education does not cover these topics and this information can be highly localized.
Purpose: We organized the Guide to Essentials in Medicine (GEM) to provide a living document that serves as a hive mind of crowdsourced logistics, policies, and wisdom for hospitalists at our institution. Additionally, this work is being expanded to develop the GEM LLM, which can reference information to answer questions providers have regarding logistics of care and how the hospital functions.
Description: A living OneNote document (Microsoft corporation) composed of over 300 pages called the GEM was created by a team of 6 hospitalist clinician editors with input from over 50 local hospitalists. Input was also obtained from specialists, care managers, nurses and community resources (e.g. local jails and detoxification centers) for policies and best practices. Continual updates are made with any new practice change or upon crowdsourced recommendations from the hospitalist staff.Pre and 1-year post implementation surveys were sent to practicing hospitalists. 39 providers responded to the pre-survey and 35 providers responded to the post-survey. The likelihood to recommend current resources for practical daily logistics in hospital internal medicine improved from a net promoter score (NPS) of 5.9 in the pre-survey to a post-survey NPS of 7.6 (p=0.0085). For providers who onboarded within the past 2 years (6 providers pre- implementation and 12 provider post-implementation), the mean NPS pre-intervention was 6.5 and improved to 8 post intervention (p=0.32). The post-survey mean usefulness (on a scale of 1-10) rating of the GEM was 7.7 (n =35, SD=2.1).The GEM was then built into a custom configured LLM (base model OpenAI GPT 5) to evaluate accuracy in answering a representative sample of 30 questions providers may ask. Responses were rated from 0-2 across domains of correctness, completeness, citation fidelity and safety risk by two practicing hospitalists for a score of 8 possible points. The average answer score was 6.2 (SD 1.9). There were numerous errors in responses noted such as hallucinating a telephone number for a homeless shelter, hallucinating a specific hospital service that does not exist, indicating the wrong type of contact precautions, and incorrectly describing the official 72-hour hold process. Given these limitations, we are working to improve accuracy and withholding LLM implementation at this time.
Conclusions: Hospitalists can effectively work together to collate wisdom into a living guide for daily practice logistics. The GEM implementation was associated with significantly higher likelihood to recommend current resources for practical daily logistics in hospital internal medicine at our large, academic medical center. However, utilizing this document with an LLM to answer questions has not yet achieved sufficient reliability (2). Further work is ongoing to improve accuracy and reliability so that an AI companion for daily practice logistics in hospital medicine can be implemented.