Purpose: Generally, in MRI, PET, and CT image datasets, more information is available for defining the target volume (or normal structures) than is used during the target segmentation. We introduce a method to take advantage of all the imaging information available for target segmentation, including multi‐modality images or multiple image sets from the same modality. Method and Materials: We generalized the multi‐valued level set deformable model (Chan et al., JVCI (2000) 11:130–141) for simultaneous 2D/3D segmentation/registration of multi‐modality images consisting of a combination of PET, CT, or MR datasets. Information from multi modality image sets is combined to define the final target volume. The method was evaluated on three patient cases, including: a non‐small cell lung cancer case with PET/CT, a cervix cancer case with PET/CT, and a prostate patient case with CT and MR. Results: In the case of the lung tumor the level set algorithm took 120 iterations for convergence, while in the case cervix tumor it converged after 30 iterations because the tumor has a deformed circular shape. In the prostate case, it took 50 iterations to converge and the results were made more sensitive to the shape prior information, because MR provides less gradient strength than PET. The computational time was on the order of few seconds in all cases. Conclusion: We a developed a new target segmentation algorithm which uses information simultaneously from multiple modalities. Our initial results indicate that the algorithm is promising and could provide physicians with a reliable contouring tool for lung, cervical, and prostate cancer. This research was partially supported by NIH grant R01 CA85181 and a grant from TomoTherapy, Inc.