Purpose: To develop a framework of 3D and 4D deformable image registration and investigate its applications in image guided radiotherapy (IGRT). Method and Materials: We have developed 3D and 4D deformable image registration methods to register fan beam CT (FBCT) and cone beam CT (CBCT) images. Applications of the 3D and 4D deformable registration include: (1) Inter‐subject atlas‐based image segmentation to automatically contour normal tissue regions of interest (ROI) in treatment planning CT images; (2) Intra‐subject atlas‐based image segmentation to delineate daily ROIs by registering daily FBCT or CBCT images to treatment planning CT images. This method is an ideal ROI delineation method for online plan adjustments; (3) 4D deformable image registration to extract average voxel trajectories for real‐time tumor tracking and for 4D tomotherapy treatment planning; (4) Deformable fusion of 4D‐PET images, which is based on deformable registration of 4D‐CT images, to minimize motion blurring in PET images without increase of committed doses or data acquisition time. Results: The deformable image registration system was tested using clinical images. Registrations of inter‐ and intra‐subject images show acceptable results. The corresponding results are reported in our supporting document. Quantitative validation of intra‐subject atlas‐based image segmentation showed agreements with physicians' manual contours. 4D registration yields voxel trajectories smooth in both spatial and temporal spaces. Problems were noticed in the cases where the registration image pairs are intrinsically different, such as gas bubbles in rectums, or deformations are extremely large, such as empty vs. full bladders. Conclusions: We have developed a general framework of applying 3D & 4D deformable image registration in image guided radiotherapy. Our clinical applications showed the positive and convincing results.