Objective assessment of operator performance during ultrasound-guided procedures

David M. Tabriz, Mandie Street, Thomas K. Pilgram, James R. Duncan

Research output: Contribution to journalArticlepeer-review

6 Scopus citations


Purpose: Simulation permits objective assessment of operator performance in a controlled and safe environment. Image-guided procedures often require accurate needle placement, and we designed a system to monitor how ultrasound guidance is used to monitor needle advancement toward a target. The results were correlated with other estimates of operator skill. Methods: The simulator consisted of a tissue phantom, ultrasound unit, and electromagnetic tracking system. Operators were asked to guide a needle toward a visible point target. Performance was video-recorded and synchronized with the electromagnetic tracking data. A series of algorithms based on motor control theory and human information processing were used to convert raw tracking data into different performance indices. Scoring algorithms converted the tracking data into efficiency, quality, task difficulty, and targeting scores that were aggregated to create performance indices. After initial feasibility testing, a standardized assessment was developed. Operators (N = 12) with a broad spectrum of skill and experience were enrolled and tested. Overall scores were based on performance during ten simulated procedures. Prior clinical experience was used to independently estimate operator skill. Results: When summed, the performance indices correlated well with estimated skill. Operators with minimal or no prior experience scored markedly lower than experienced operators. The overall score tended to increase according to operator's clinical experience. Operator experience was linked to decreased variation in multiple aspects of performance. The aggregated results of multiple trials provided the best correlation between estimated skill and performance. A metric for the operator's ability to maintain the needle aimed at the target discriminated between operators with different levels of experience. Conclusions: This study used a highly focused task model, standardized assessment, and objective data analysis to assess performance during simulated ultrasound-guided needle placement. The performance indices were closely related to operator experience.

Original languageEnglish
Pages (from-to)641-652
Number of pages12
JournalInternational Journal of Computer Assisted Radiology and Surgery
Issue number5
StatePublished - Sep 2011


  • Assessment
  • Education
  • Electromagnetic tracking
  • Information theory
  • Simulation


Dive into the research topics of 'Objective assessment of operator performance during ultrasound-guided procedures'. Together they form a unique fingerprint.

Cite this