Objective: Cone beam CT (CBCT) images contain more scatter than a conventional CT image and therefore provide inaccurate Hounsfield units (HU). Consequently CBCT images cannot be used directly for radiotherapy dose calculation. The aim of this study is to enable dose calculations to be performed with the use of CBCT images taken during radiotherapy and evaluate the necessity of re-planning. Methods: A phantom, a standard prostate cancer patient and prostate cancer patients with single and double metallic hips were imaged using both CT and CBCT. A multilevel threshold algorithm (MLT) was used to categorise pixel values in the CBCT images into segments of homogeneous HU. The variation in HU with position in the CBCT images was taken into consideration and the benefit of using a large number of materials has been explored. This segmentation method relies upon the operator dividing the CBCT data into a set of volumes where the variation in the relationship between pixel values and HUs is small. In addition, an automated MLT algorithm was developed to reduce the operator time associated with the process. Furthermore, magnetic resonance (MR) images of the standard prostate case were segmented and converted into HUs using the MLT algorithm. Radiotherapy treatment plans were generated from CT images and then copied to the segmented CBCT and MR data sets and the doses were recalculated and compared using pencil beam (PB), collapsed cone (CC) and Monte Carlo (MC) algorithms. Results: Compared with the planning CT (pCT) treatment plan, in the phantom case, a gamma evaluation showed all points in planning target volume (PTV), rectum and bladder had gamma value < 1 (3%/3 mm) in the segmented CBCT, when considering only 2 material bins, water and bone. For the standard patient case, using 3 materials, air, water and bone, was accurate enough to provide accurate dose calculations with differences of less than 2%. For the patient with a metallic hip, increasing the number of bins to define the material type from 7 materials to 8 materials, required 50% more operator time to improve the accuracy by 0.01% using PB and CC and 0.05% when using MC algorithms. The use of 5 values of HU (air, adipose, water, bone and metal implant) gave the best balance between dose accuracy and operator time (3.5 hours). For the patient with double hip prosthetics, segmenting CBCT into 5 materials with the MLT algorithm showed-0.46% dose difference with 8 hours operator time, whilst the automated MLT algorithm showed-1.36%. For the standard case, the segmentation of MR images, into 3 materials, resulted in a dose difference of-1.31% with 2 hours operator time.