In this paper, a method that detects an image-based respiratory signal automatically in Cone Beam Computed Tomography (CBCT) projection datasets is proposed. The proposed Intensity Flow Dimensionality Reduction method (IFDR) uses optical flow tracking to estimate a set of dense intensity flow vectors from every adjacent pair of projections in the projection dataset. A dimensionality reduction method is applied to the intensity flow vectors to distil them into an eigensystem in which the first few principal components (up to 3 in this work) are combined to represent the motion patterns in the dataset. The algorithm was experimentally evaluated on clinical patient datasets. The extracted respiratory signal using IFDR was compared to respiratory signals measured using 1) the diaphragm position and 2) a trajectory of fiducial markers implanted in and near the tumor. IFDR-based respiratory signal showed an average phase shift of 3.8 ± 1.9 projections (0.35% of the projection set) comparing to the diaphragm position-based signal, and an average phase shift of 3.59 ± 2.44 projections (0.15% of the projection set) comparing to the internal markers-based signal. IFDR was able to extract the respiratory signal in all projections of all the patients' dataset without using any external devices, internal markers or requiring any structure such as the diaphragm to be visible in the CBCT projections. This respiratory signal extracted correlates to the tumor position since the motion was estimated from the soft tissues in and around the tumor.