Purpose: To illustrate a method for quantifying systematic and random three-dimensional (3D) segmentation uncertainties for an anatomical structure of arbitrary shape within a population of patients imaged by cone-beam computed tomography (CBCT). Methods: Prostate, bladder and rectum of four prostate cancer patients were independently delineated by five observers on CBCT and fan-beam computed tomography (FBCT) images. A surface mesh-based method was used to calculate the surface-to-surface perpendicular distances as a function of location on the organ surface. The intra-observer and the inter-observer segmentation uncertainties were computed and mapped to a structure population model which was constructed by averaging segmentation uncertainties quantified for each individual patient. The surface location dependent-systematic segmentation errors were calculated based on the mean differences between the surfaces drawn on the CBCT and those on the FBCT. Results: Quantification of segmentation uncertainties based on a 3D population model was successfully performed. Observer-dependent uncertainties were largest in areas of abutting organs. Due to uncertainties of deformable image registration in the pelvis, systematic differences caused by imaging modalities were only calculated for the prostate. Large imaging modality-introduced systematic errors were identified in the posterior prostatic apex and the vicinity with seminal vesicles. Conclusions: A population-based delineation uncertainty model has been established for pelvic CBCT that allows assessment of manual contouring uncertainties. This information should be considered for definition of action levels for CBCT-based image guidance as well as uncertainty-weighted deformable image registration and dose summation in adaptive radiotherapy.