Evaluation of crowd-sourced assessment of the critical view of safety in laparoscopic cholecystectomy

Shanley B. Deal, Dimitrios Stefanidis, Dana Telem, Robert D. Fanelli, Marian McDonald, Michael Ujiki, L. Michael Brunt, Adnan A. Alseidi

Research output: Contribution to journalArticlepeer-review

32 Scopus citations


Background: Faculty experts (FE) and crowd workers (CW) can assess technical skill, but assessment of operative technique has not been explored. We sought to evaluate if CW could be taught to assess completion of the critical view of safety (CVS) in laparoscopic cholecystectomy. Methods: We prepared 160 blinded, surgical videos of laparoscopic cholecystectomy from public domain websites. Videos were edited to ≤60 s, ending when a structure was cut/clipped. CW analyzed videos using Global Objective Assessment of Laparoscopic Skills (GOALS) and CVS criteria assessment tools after watching an instructional tutorial. Ten videos were randomly selected from each performance quartile based on GOALS. Five FE rated the 40 videos using GOALS and CVS. Linear mixed effects models derived average CW and FE ratings for GOALS and CVS for each video. Spearman correlation coefficients (SCC) were used to assess the degree of correlation between performance measures. Satisfactory completion of the CVS was defined as scoring an average CVS ≥ 5. Videos with an average GOALS ≥ 15 were considered top technical performers. Results: A high degree of correlation was seen between all performance measures: CVS ratings between CW and FE, SCC 0.89 (p < 0.001); GOALS and CVS ratings SCC 0.77 (p < 0.001) for CW, and SCC 0.71 (p < 0.001) for FE. Sixteen videos were assigned top technical performer ratings by both CW and FE but the average CVS was inadequate (3.8 and 3.6, respectively), and the percentage of satisfactory CVS ≥ 5 was 12.5%. Conclusions: A high degree of correlation was found between CW and FE in assessment of the CVS. However, in this video analysis, high technical performers did not achieve a complete CVS in most cases. Educating CW to assess operative technique for the identification of low or average performers is feasible and may broaden the application of this assessment and feedback tool.

Original languageEnglish
Pages (from-to)5094-5100
Number of pages7
JournalSurgical endoscopy
Issue number12
StatePublished - Dec 1 2017


  • Critical view of safety
  • Crowd-sourced assessment of technical skill
  • Laparoscopic cholecystectomy
  • Video-based education


Dive into the research topics of 'Evaluation of crowd-sourced assessment of the critical view of safety in laparoscopic cholecystectomy'. Together they form a unique fingerprint.

Cite this