Facial expression monitoring system for predicting patient’s sudden movement during radiotherapy using deep learning

Kwang Hyeon Kim, Kyeongyun Park, Haksoo Kim, Byungdu Jo, Sang Hee Ahn, Chankyu Kim, Myeongsoo Kim, Tae Ho Kim, Se Byeong Lee, Dongho Shin, Young Kyung Lim, Jong Hwi Jeong

Research output: Contribution to journalArticlepeer-review

12 Scopus citations


Purpose: Imaging, breath-holding/gating, and fixation devices have been developed to minimize setup errors so that the prescribed dose can be exactly delivered to the target volume in radiotherapy. Despite these efforts, additional patient monitoring devices have been installed in the treatment room to view patients’ whole-body movement. We developed a facial expression recognition system using deep learning with a convolutional neural network (CNN) to predict patients’ advanced movement, enhancing the stability of the radiation treatment by giving warning signs to radiation therapists. Materials and methods: Convolutional neural network model and extended Cohn-Kanade datasets with 447 facial expressions of source images for training were used. Additionally, a user interface that can be used in the treatment control room was developed to monitor real-time patient's facial expression in the treatment room, and the entire system was constructed by installing a camera in the treatment room. To predict the possibility of patients' sudden movement, we categorized facial expressions into two groups: (a) uncomfortable expressions and (b) comfortable expressions. We assumed that the warning sign about the sudden movement was given when the uncomfortable expression was recognized. Results: We have constructed the facial expression monitoring system, and the training and test accuracy were 100% and 85.6%, respectively. In 10 patients, their emotions were recognized based on their comfortable and uncomfortable expressions with 100% detection rate. The detected various emotions were represented by a heatmap and motion prediction accuracy was analyzed for each patient. Conclusion: We developed a system that monitors the patient's facial expressions and predicts patient's advanced movement during the treatment. It was confirmed that our patient monitoring system can be complementarily used with the existing monitoring system. This system will help in maintaining the initial setup and improving the accuracy of radiotherapy for the patients using deep learning in radiotherapy.

Original languageEnglish
Pages (from-to)191-199
Number of pages9
JournalJournal of applied clinical medical physics
Issue number8
StatePublished - Aug 1 2020


  • Radiotherapy
  • convolutional neural network
  • facial expression recognition
  • patient monitoring system
  • predicting body movement


Dive into the research topics of 'Facial expression monitoring system for predicting patient’s sudden movement during radiotherapy using deep learning'. Together they form a unique fingerprint.

Cite this