Background:

Handoffs are a core competency of hospitalists. While the Society of Hospital Medicine and others recommend improving handoffs, there is a lack of validated tools for teaching and assessing handoff quality. The specific aims of this project were to create video–based examples of varying levels of handoff performance for education and to validate an assessment instrument, the Hand–off CEX.

Methods:

Six video–based scenarios were developed highlighting varying levels of performance in the domains of communication, professionalism and setting. Each video permuted one domain of performance while holding the others constant. Scripts were based upon formal clinical observations. Videos ranged in length from 3–5 minutes. Faculty were recruited via email to pilot test the videos and for instrument validation. When watching videos in random order, faculty were instructed to use the Hand–off CEX to rate performance. The Handoff CEX was developed as a paper–based instrument in which individuals are rated in four domains from unsatisfactory [1] to superior [9] with qualitative anchors defining performance level. After rating videos, faculty identified barriers and facilitators to the handoff. Descriptive statistics and two tests of reliability, Cronbach’s alpha and Kendall’s coefficient of concordance, were performed. Two–way ANOVA was performed to examine for rater bias.

Results:

Fourteen faculty provided 73 of 90 (82%) possible ratings. Overall hand–off quality ratings for the gold standard video ranged from 7–9 with a mean of 8.5 (SD 0.7). Mean ratings for the domains of professionalism, setting and communication showed a statistically significant difference between unsatisfactory and superior (P<0.001 for all). While raters could distinguish all levels of professionalism, they had difficulty distinguishing between satisfactory and superior communication (see Figures 1 and 2). Two–way ANOVA revealed no evidence of rater bias. Reliability testing revealed a Cronbach’s alpha of 0.81 (0.8 = optimal) and Kendall’s coefficient of concordance of 0.59 (>0.6 = high reliability). Faculty participants noted the face validity of video scenarios, specifically those portraying setting and communication skills.

Conclusions:

Although the 3–item CEX is a reliable and valid tool to assess varying levels of videos depicting handoff performance, raters have difficulty distinguishing between satisfactory and superior levels. Future work will aim to use the 3–item Hand–off CEX in clinical situations.

Figure 1Faculty Ratings of Professionalism by Performance.

Figure 2Faculty Ratings of Communication by Performance.