Background: Whiteboard mini lectures (WMLs), a “chalk talk” subtype, are interactive, small group sessions that use a whiteboard and markers as the primary teaching tools, A distinguishing feature of WMLs is the flexibility to creatively teach material by combining written text with novel, hand-drawn visuals such as diagrams, flow charts, and graphics. Assessment of WMLs has relevance to academic hospitalists, yet there is an absence of a specific instrument to evaluate these teaching sessions and provide formative feedback to the instructor.
Purpose: To develop and collect validity evidence for The OWL (Optimizing Whiteboard Lectures) Checklist, a novel WML direct observation instrument. The purpose of the OWL Checklist is to provide residents, fellows, and faculty with standardized formative feedback for improving their whiteboard teaching.
Description: The authors performed a literature review for medical and non-medical frameworks of effective whiteboard teaching skills and for existing clinical teaching and lecture assessment tools. Two authors (REN and DNR) with expertise in WMLs extracted data pertinent to whiteboard-based teaching. We organized teaching skills pertinent to WMLs into broad domains and sought consensus from a diverse group of medical educators. After 2 rounds of revisions, we reached consensus on 5 domains of WML teaching skills. We then created behavioral elements for each domain that are directly observable throughout a WML. Following 3 rounds of review among our group of medical educators and 3 further rounds of review from 2 medical education experts, we achieved consensus on the preliminary instrument. The OWL Checklist comprises 21 behavioral elements organized within 5 domains: (1) learning environment, (2) whiteboard design, (3) learner engagement, (4) promotes understanding and retention, and (5) global measure of teaching performance. Each item is worth 1 point based on the binary outcome of “observed” or “not observed,” for a total possible score of 24.
Conclusions: We pilot tested the OWL Checklist with residents, fellows, and attending physicians from March-June 2023 at Beth Israel Deaconess Medical Center. The average score was 18.5 (SD: 3.88). Based on a power calculation, we determined that we need to perform at least 58 WML observations to detect the smallest possible difference between teacher scores (1 point). We recruited a group of raters who received a faculty development session reviewing logistics of the OWL Checklist. Throughout the 2023-2024 academic year, we will employ Kane’s four-stage model (scoring, generalization, extrapolation, and implications) to collect and analyze validity evidence for the OWL Checklist. We anticipate having sufficient evaluations by SHM Converge 2024 to run the kappa statistic to evaluate inter-rater reliability (generalization) and mixed linear regression to compare resident, fellow, and attending physician scores (extrapolation). To our knowledge, The OWL Checklist is the first assessment tool for evaluating WMLs.
