Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth From Ten Pediatrics Residency Programs

Yoon Soo Park, Patricia J. Hicks, Carol Carraccio, Melissa J. Margolis, Alan Schwartz, Nick Potisek, Allison McBride, Kathleen M. Donnelly, Meredith Carter, Teri Turner, Renuka Verma, Su Ting T. Li, Amanda Osta, Hilary M. Haftel, Lynn Thoreson, Linda Waggoner-Fountain, Mark Mendelsohn, Ann Burke, Brian Clauser, Thomas Rebbecchi

Research output: Contribution to journalConference articlepeer-review

Abstract

Purpose: This study investigates the impact of incorporating observer-reported workload into workplace-based assessment (WBA) scores on (1) psychometric characteristics of WBA scores and (2) measuring changes in performance over time using workload-unadjusted versus workload-adjusted scores.

Method: Structured clinical observations and multisource feedback instruments were used to collect WBA data from first-year pediatrics residents at 10 residency programs between July 2016 and June 2017. Observers completed items in 8 subcompetencies associated with Pediatrics Milestones. Faculty and resident observers assessed workload using a sliding scale ranging from low to high; all item scores were rescaled to a 1-5 scale to facilitate analysis and interpretation. Workload-adjusted WBA scores were calculated at the item level using three different approaches, and aggregated for analysis at the competency level. Mixed-effects regression models were used to estimate variance components. Longitudinal growth curve analyses examined patterns of developmental score change over time.

Results: On average, participating residents (n = 252) were assessed 5.32 times (standard deviation = 3.79) by different raters during the data collection period. Adjusting for workload yielded better discrimination of learner performance, and higher reliability, reducing measurement error by 28%. Projections in reliability indicated needing up to twice the number of raters when workload-unadjusted scores were used. Longitudinal analysis showed an increase in scores over time, with significant interaction between workload and time; workload also increased significantly over time.

Conclusions: Incorporating a measure of observer-reported workload could improve the measurement properties and the ability to interpret WBA scores.
Original languageAmerican English
Pages (from-to)S21-S29
JournalJournal of the Association of American Medical Colleges
Volume93
Issue number11S
DOIs
StatePublished - Nov 2018
Event57th Annual Research in Medical Education - Austin, United States
Duration: Nov 2 2018Nov 6 2018
Conference number: 57

Keywords

  • Clincial Competence*
  • Educational Measurement
  • Humans
  • Internship and Residency*
  • Pediatrics/education*
  • Psychometrics
  • Workload*

Disciplines

  • Pediatrics
  • Medical Education

Cite this