Interactive training and operation ecosystem for surgical tasks in mixed reality

Ehsan Azimi; Camilo Molina; Alexander Chang; Judy Huang; Chien Ming Huang; Peter Kazanzides

doi:10.1007/978-3-030-01201-4_3

Interactive training and operation ecosystem for surgical tasks in mixed reality

Ehsan Azimi, Camilo Molina, Alexander Chang, Judy Huang, Chien Ming Huang, Peter Kazanzides

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

Inadequate skill in performing surgical tasks can lead to medical errors and cause avoidable injury or death to the patients. On the other hand, there are situations where a novice surgeon or resident does not have access to an expert while performing a task. We therefore propose an interactive ecosystem for both training and practice of surgical tasks in mixed reality, which consists of authoring of the desired surgical task, immersive training and practice, assessment of the trainee, and remote coaching and analysis. This information-based ecosystem will also provide the data to train machine learning algorithms. Our interactive ecosystem involves a head-mounted display (HMD) application that can provide feedback as well as audiovisual assistance for training and live clinical performance of the task. In addition, the remote monitoring station provides the expert with a real-time view of the scene from the user’s perspective and enables guidance by providing annotation directly on the user’s scene. We use bedside ventriculostomy, a neurosurgical procedure, as our illustrative use case; however the modular design of the system makes it expandable to other procedures.

Original language	English
Title of host publication	OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018
Editors	Anand Malpani, Marco A. Zenati, Cristina Oyarzun Laura, M. Emre Celebi, Duygu Sarikaya, Noel C. Codella, Allan Halpern, Marius Erdt, Lena Maier-Hein, Luo Xiongbiao, Stefan Wesarg, Danail Stoyanov, Zeike Taylor, Klaus Drechsler, Kristin Dana, Anne Martel, Raj Shekhar, Sandrine De Ribaupierre, Tobias Reichl, Jonathan McLeod, Miguel Angel González Ballester, Toby Collins, Marius George Linguraru
Publisher	Springer Verlag
Pages	20-29
Number of pages	10
ISBN (Print)	9783030012007
DOIs	https://doi.org/10.1007/978-3-030-01201-4_3
State	Published - 2018
Event	1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018 - Granada, Spain Duration: Sep 16 2018 → Sep 20 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11041 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018
Country/Territory	Spain
City	Granada
Period	09/16/18 → 09/20/18

Keywords

Artificial intelligence
Medical augmented reality
Surgical simulation and modeling
Surgical training and assessment

Access to Document

10.1007/978-3-030-01201-4_3

Cite this

Azimi, E., Molina, C., Chang, A., Huang, J., Huang, C. M., & Kazanzides, P. (2018). Interactive training and operation ecosystem for surgical tasks in mixed reality. In A. Malpani, M. A. Zenati, C. Oyarzun Laura, M. E. Celebi, D. Sarikaya, N. C. Codella, A. Halpern, M. Erdt, L. Maier-Hein, L. Xiongbiao, S. Wesarg, D. Stoyanov, Z. Taylor, K. Drechsler, K. Dana, A. Martel, R. Shekhar, S. De Ribaupierre, T. Reichl, J. McLeod, M. A. González Ballester, T. Collins, ... M. G. Linguraru (Eds.), OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018 (pp. 20-29). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11041 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-01201-4_3

Azimi, Ehsan ; Molina, Camilo ; Chang, Alexander et al. / Interactive training and operation ecosystem for surgical tasks in mixed reality. OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. editor / Anand Malpani ; Marco A. Zenati ; Cristina Oyarzun Laura ; M. Emre Celebi ; Duygu Sarikaya ; Noel C. Codella ; Allan Halpern ; Marius Erdt ; Lena Maier-Hein ; Luo Xiongbiao ; Stefan Wesarg ; Danail Stoyanov ; Zeike Taylor ; Klaus Drechsler ; Kristin Dana ; Anne Martel ; Raj Shekhar ; Sandrine De Ribaupierre ; Tobias Reichl ; Jonathan McLeod ; Miguel Angel González Ballester ; Toby Collins ; Marius George Linguraru. Springer Verlag, 2018. pp. 20-29 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{b7f60e6cc68f4039a375a35b3104e1aa,

title = "Interactive training and operation ecosystem for surgical tasks in mixed reality",

abstract = "Inadequate skill in performing surgical tasks can lead to medical errors and cause avoidable injury or death to the patients. On the other hand, there are situations where a novice surgeon or resident does not have access to an expert while performing a task. We therefore propose an interactive ecosystem for both training and practice of surgical tasks in mixed reality, which consists of authoring of the desired surgical task, immersive training and practice, assessment of the trainee, and remote coaching and analysis. This information-based ecosystem will also provide the data to train machine learning algorithms. Our interactive ecosystem involves a head-mounted display (HMD) application that can provide feedback as well as audiovisual assistance for training and live clinical performance of the task. In addition, the remote monitoring station provides the expert with a real-time view of the scene from the user{\textquoteright}s perspective and enables guidance by providing annotation directly on the user{\textquoteright}s scene. We use bedside ventriculostomy, a neurosurgical procedure, as our illustrative use case; however the modular design of the system makes it expandable to other procedures.",

keywords = "Artificial intelligence, Medical augmented reality, Surgical simulation and modeling, Surgical training and assessment",

author = "Ehsan Azimi and Camilo Molina and Alexander Chang and Judy Huang and Huang, {Chien Ming} and Peter Kazanzides",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2018. Copyright: Copyright 2018 Elsevier B.V., All rights reserved.; 1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018 ; Conference date: 16-09-2018 Through 20-09-2018",

year = "2018",

doi = "10.1007/978-3-030-01201-4_3",

language = "English",

isbn = "9783030012007",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "20--29",

editor = "Anand Malpani and Zenati, {Marco A.} and {Oyarzun Laura}, Cristina and Celebi, {M. Emre} and Duygu Sarikaya and Codella, {Noel C.} and Allan Halpern and Marius Erdt and Lena Maier-Hein and Luo Xiongbiao and Stefan Wesarg and Danail Stoyanov and Zeike Taylor and Klaus Drechsler and Kristin Dana and Anne Martel and Raj Shekhar and {De Ribaupierre}, Sandrine and Tobias Reichl and Jonathan McLeod and {Gonz{\'a}lez Ballester}, {Miguel Angel} and Toby Collins and Linguraru, {Marius George}",

booktitle = "OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018",

}

Azimi, E, Molina, C, Chang, A, Huang, J, Huang, CM & Kazanzides, P 2018, Interactive training and operation ecosystem for surgical tasks in mixed reality. in A Malpani, MA Zenati, C Oyarzun Laura, ME Celebi, D Sarikaya, NC Codella, A Halpern, M Erdt, L Maier-Hein, L Xiongbiao, S Wesarg, D Stoyanov, Z Taylor, K Drechsler, K Dana, A Martel, R Shekhar, S De Ribaupierre, T Reichl, J McLeod, MA González Ballester, T Collins & MG Linguraru (eds), OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11041 LNCS, Springer Verlag, pp. 20-29, 1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018, Granada, Spain, 09/16/18. https://doi.org/10.1007/978-3-030-01201-4_3

Interactive training and operation ecosystem for surgical tasks in mixed reality. / Azimi, Ehsan; Molina, Camilo; Chang, Alexander et al.
OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. ed. / Anand Malpani; Marco A. Zenati; Cristina Oyarzun Laura; M. Emre Celebi; Duygu Sarikaya; Noel C. Codella; Allan Halpern; Marius Erdt; Lena Maier-Hein; Luo Xiongbiao; Stefan Wesarg; Danail Stoyanov; Zeike Taylor; Klaus Drechsler; Kristin Dana; Anne Martel; Raj Shekhar; Sandrine De Ribaupierre; Tobias Reichl; Jonathan McLeod; Miguel Angel González Ballester; Toby Collins; Marius George Linguraru. Springer Verlag, 2018. p. 20-29 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11041 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Interactive training and operation ecosystem for surgical tasks in mixed reality

AU - Azimi, Ehsan

AU - Molina, Camilo

AU - Chang, Alexander

AU - Huang, Judy

AU - Huang, Chien Ming

AU - Kazanzides, Peter

PY - 2018

Y1 - 2018

N2 - Inadequate skill in performing surgical tasks can lead to medical errors and cause avoidable injury or death to the patients. On the other hand, there are situations where a novice surgeon or resident does not have access to an expert while performing a task. We therefore propose an interactive ecosystem for both training and practice of surgical tasks in mixed reality, which consists of authoring of the desired surgical task, immersive training and practice, assessment of the trainee, and remote coaching and analysis. This information-based ecosystem will also provide the data to train machine learning algorithms. Our interactive ecosystem involves a head-mounted display (HMD) application that can provide feedback as well as audiovisual assistance for training and live clinical performance of the task. In addition, the remote monitoring station provides the expert with a real-time view of the scene from the user’s perspective and enables guidance by providing annotation directly on the user’s scene. We use bedside ventriculostomy, a neurosurgical procedure, as our illustrative use case; however the modular design of the system makes it expandable to other procedures.

AB - Inadequate skill in performing surgical tasks can lead to medical errors and cause avoidable injury or death to the patients. On the other hand, there are situations where a novice surgeon or resident does not have access to an expert while performing a task. We therefore propose an interactive ecosystem for both training and practice of surgical tasks in mixed reality, which consists of authoring of the desired surgical task, immersive training and practice, assessment of the trainee, and remote coaching and analysis. This information-based ecosystem will also provide the data to train machine learning algorithms. Our interactive ecosystem involves a head-mounted display (HMD) application that can provide feedback as well as audiovisual assistance for training and live clinical performance of the task. In addition, the remote monitoring station provides the expert with a real-time view of the scene from the user’s perspective and enables guidance by providing annotation directly on the user’s scene. We use bedside ventriculostomy, a neurosurgical procedure, as our illustrative use case; however the modular design of the system makes it expandable to other procedures.

KW - Artificial intelligence

KW - Medical augmented reality

KW - Surgical simulation and modeling

KW - Surgical training and assessment

UR - http://www.scopus.com/inward/record.url?scp=85054886916&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01201-4_3

DO - 10.1007/978-3-030-01201-4_3

M3 - Conference contribution

AN - SCOPUS:85054886916

SN - 9783030012007

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 20

EP - 29

BT - OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018

A2 - Malpani, Anand

A2 - Zenati, Marco A.

A2 - Oyarzun Laura, Cristina

A2 - Celebi, M. Emre

A2 - Sarikaya, Duygu

A2 - Codella, Noel C.

A2 - Halpern, Allan

A2 - Erdt, Marius

A2 - Maier-Hein, Lena

A2 - Xiongbiao, Luo

A2 - Wesarg, Stefan

A2 - Stoyanov, Danail

A2 - Taylor, Zeike

A2 - Drechsler, Klaus

A2 - Dana, Kristin

A2 - Martel, Anne

A2 - Shekhar, Raj

A2 - De Ribaupierre, Sandrine

A2 - Reichl, Tobias

A2 - McLeod, Jonathan

A2 - González Ballester, Miguel Angel

A2 - Collins, Toby

A2 - Linguraru, Marius George

PB - Springer Verlag

T2 - 1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018

Y2 - 16 September 2018 through 20 September 2018

ER -

Azimi E, Molina C, Chang A, Huang J, Huang CM, Kazanzides P. Interactive training and operation ecosystem for surgical tasks in mixed reality. In Malpani A, Zenati MA, Oyarzun Laura C, Celebi ME, Sarikaya D, Codella NC, Halpern A, Erdt M, Maier-Hein L, Xiongbiao L, Wesarg S, Stoyanov D, Taylor Z, Drechsler K, Dana K, Martel A, Shekhar R, De Ribaupierre S, Reichl T, McLeod J, González Ballester MA, Collins T, Linguraru MG, editors, OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. Springer Verlag. 2018. p. 20-29. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-01201-4_3

Interactive training and operation ecosystem for surgical tasks in mixed reality

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this