Large-language-model empowered 3D dose prediction for intensity-modulated radiotherapy

Zehao Dong, Yixin Chen, Hiram Gay, Yao Hao, Geoffrey D. Hugo, Pamela Samson, Tianyu Zhao

Research output: Contribution to journalArticlepeer-review

Abstract

Background: Treatment planning is currently a patient specific, time-consuming, and resource demanding task in radiotherapy. Dose-volume histogram (DVH) prediction plays a critical role in automating this process. The geometric relationship between DVHs in radiotherapy plans and organs-at-risk (OAR) and planning target volume (PTV) has been well established. This study explores the potential of deep learning models for predicting DVHs using images and subsequent human intervention facilitated by a large-language model (LLM) to enhance the planning quality. Method: We propose a pipeline to convert unstructured images to a structured graph consisting of image-patch nodes and dose nodes. A novel Dose Graph Neural Network (DoseGNN) model is developed for predicting DVHs from the structured graph. The proposed DoseGNN is enhanced with the LLM to encode massive knowledge from prescriptions and interactive instructions from clinicians. In this study, we introduced an online human-AI collaboration (OHAC) system as a practical implementation of the concept proposed for the automation of intensity-modulated radiotherapy (IMRT) planning. Results: The proposed DoseGNN model was compared to widely employed DL models used in radiotherapy, including Swin Transformer, 3D U-Net CNN, and vanilla MLP. For PTV, DoseGNN achieved the mean absolute error (MAE) of (Formula presented.), (Formula presented.), (Formula presented.), and (Formula presented.) between true plans and predicted plans that were 64%, 53%, 64%, 61% of the best baseline model. For the worst case among OARs (left lung, right lung, chest wall, heart, spinal cord), DoseGNN achieved the mean absolute error of (Formula presented.), (Formula presented.), (Formula presented.) that were 85%, 91%, 80% of the best baseline model. Moreover, the LLM-empowered DoseGNN model facilitates seamless adjustment to treatment plans through interaction with clinicians using natural language. Conclusion: We developed DoseGNN, a novel deep learning model for predicting delivered radiation doses from medical images, enhanced by LLM to allow adjustment through seamless interaction with clinicians. The preliminary results confirm DoseGNN's superior accuracy in DVH prediction relative to typical DL methods, highlighting its potential to facilitate an online clinician-AI collaboration system for streamlined treatment planning automation.

Original languageEnglish
Pages (from-to)619-632
Number of pages14
JournalMedical physics
Volume52
Issue number1
DOIs
StatePublished - Jan 2025

Keywords

  • dose-volume histogram (DVH) prediction
  • graph neural networks
  • large language models

Fingerprint

Dive into the research topics of 'Large-language-model empowered 3D dose prediction for intensity-modulated radiotherapy'. Together they form a unique fingerprint.

Cite this