Measurement Principles in Medical Education: Validity, Reliability, and Utility, Oh My!

As program directors, we strive to assess our trainees in a thoughtful and fair manner, both for their benefit and that of the children who rely on their competence. To be able to select appropriate assessment tools that measure aspects of performance that are meaningful, one must have an informed understanding of strengths and weaknesses of various assessment tools and techniques. Further, it is helpful to understand concepts about measurement that have practical applications in the training of future pediatricians. Key concepts that need to be considered when assessing competence are validity, reliability, and the utility or usefulness of a given tool when it is used for purposes of assessment.1,2 There are multitudes of other statistical concepts and definitions that are important in activities such as analyzing research data and developing and validating assessment tools,3,4 however this chapter will not cover those topics. Practical aspects and considerations about resident assessment should be understood to enable the program director to
use and interpret assessment data in meaningful and cogent ways.5 This is particularly important because outcomes for the learner and our patients depend on our ability to meaningfully and accurately interpret
the data that we glean from our assessment tools. So, although there is much debate in the medical education literature regarding how best to assess competencies, as mentioned in Part II of this primer, several
strategies can be used to involve multiple raters from multiple venues (triangulation), to combine assessment data, and utilize assessment methods in a manner that accurately tells us “how our resident is doing”.