Development and Validation of a Model to Predict Milestone Levels Based on Entrustable Professional Activity Entrustment-Supervision Levels

Purpose – Residency programs are increasingly interested in or required to assess residents using Accreditation Council for Graduate Medical Education (ACGME) milestones and specialty-defined entrustable professional activities (EPAs). The authors aimed to develop a model to predict individual residents’ milestone levels based on their assigned EPA entrustment-supervision levels. Method – During 3 academic years from 2021 to 2024, the authors conducted a multisite prospective cohort study at 48 U.S. pediatric residency programs. Programs collected entrustment-supervision levels for the 17 general pediatrics EPAs and milestone levels for the 22 ACGME pediatric milestones for every resident biannually. EPA and milestone ratings were assigned by clinical competency committees. The first 4 of 6 biannual data reporting cycles were used to fit multilevel structural equation models and produce equations to generate, for each resident, predicted milestone levels based on EPA entrustment-supervision levels. They developed 2 models: one using 17 EPAs and one using 12 EPAs. Results – Data used for modeling represented 4, 328 residents, with 164, 886 total entrustment-supervision levels across the general pediatrics EPAs and 243, 949 total milestone levels across the pediatric milestones. The fit of the round 1 to 4 model to the round 1 to 4 data (internal prediction) was excellent for both models, with comparative fit indexes of 0.982 (17 EPAs) and 0.981 (12 EPAs). The ability of the round 1 to 4 model to predict milestones for reporting cycles 5 and 6 (external prediction) was similar to the internal predictions, with correlation coefficients of 0.68 (17 EPAs) and 0.69 (12 EPAs) for round 5 and 0.72 (17 EPAs) and 0.68 (12 EPAs) for round 6. Conclusions – This study demonstrates a strong ability to predict milestone levels based on EPA entrustment-supervision levels in a manner that enables meaningful use of EPAs and milestones in assessment efforts at residency programs.