Background: Abdominal paracentesis is a common procedure performed among patients admitted to the hospital. The current standard of care for bedside paracentesis utilizes pre-procedural (PP) ultrasound. However, real-time (RT) guidance may enhance safety and improve procedural success rates. Successful and safe execution of RT technique requires additional training and skill development. The Medicine Bedside Procedure Service (MPS) at the University of Minnesota offers an elective rotation for residents and since 2019 has performed all paracenteses using RT guidance. There is a lack of validated assessment tools for bedside procedures and no studies have explored the factors required to achieve competency using RT guidance technique.
Methods: We developed a Qualtrics-based survey tool consisting of ten simulated learner-performed paracenteses isolating the following factors; 1. Site Selection, 2. Stability of the ultrasound probe during the procedure, 3. Orientation of the probe during needle advancement, 4. Visualization of the needle throughout the procedure, and 5. Catheter deployment technique. Each of these factors was associated with a 5-point Likert scale ranging from (1) observer only to (5) Aspirational with a score of (4) corresponding to ready for unsupervised practice. Each set of factors was then associated with an overall competency assessment using the same 5-point Likert scale which asked the evaluator to incorporate all the aforementioned factors into their overall competency assessment.
Results: A total of 23 experts completed at least one video assessment leaving a sample of 203 total videos for analysis. Most experts had internal medicine training (83%), were MD or DO’s (92%), and were from the sponsoring institution (61%). Exact expert agreement across all factors and overall competency assessment was fair to moderate with k ranging from 0.24-0.53 but improved substantially when dichotomized to scores of 1/2/3 (not ready for independent practice) versus 4/5 (ready for independent practice) with k ranging from 0.62-0.94. In the final proportional odds mixed-effects model, all five factors were statistically significant predictors of the overall assessment score. Each factor was associated with increased odds of receiving a higher overall performance rating. Factor 1 was associated with 2.7 times higher odds of a better overall assessment (95% CI: 1.1–6.5). Similarly, factor 2, factor 3, factor 4, and factor 5 were associated with odds ratios of 5.2 (95% CI: 2.1–12.5), 2.5 (95% CI: 1.1–5.7), 16.0 (95% CI: 5.2–48.5), and 7.6 (95% CI: 3.2–18.0), respectively
Conclusions: Using a series of simulated videos of differing expertise and competency, we have determined the relative predictive validity between of 5 factors that correlate with progressing along a competency-based assessment tool. When considering the clinically relevant inflection point, being prepared for independent practice versus not, there was a high degree of correlation among the expert reviewers. This is an important step in establishing the validity of the tool as it reflects a clinically meaningful outcome. While our study demonstrates the tool’s ability to discriminate between high and low performers in the simulated environment, real-world validation is needed to assess its performance and fidelity in the clinical environment. As MPS become more common and their skills more developed, learner assessment tools will need to evolve to ensure acceptable levels of competency.
