Purpose: The purpose of this article is to provide a review of the competencies and associated assessment techniques relevant to the practice of anesthesiology. Although many of the competencies are difficult to define and measure specifically, advances in assessment techniques have provided more opportunities to gather meaningful performance data. Principal findings: Establishing the competence of anesthesiologists demands a host of measures, including standardized tests and less-structured peer evaluations. Simulation-based assessment will play an increasingly important role both in certification and in maintenance of certification for anesthesiologists. Conclusions: While there are many psychometric challenges associated with the assessments pertinent to the education of anesthesiologists, technological advances combined with an increased awareness of sound measurement principles will yield more meaningful competency measures that can be used to improve the practice of anesthesiology.