BackgroundTo compare the dosimetric effects of Acuros XB (AXB) and Anisotropic Analytical Algorithm (AAA) on volumetric modulated arc therapy (VMAT) planning for postoperative prostate cancer patients irradiated using an endorectal balloon (ERB).MethodsWe measured central axis doses with film in a phantom containing an air cavity, and compared measurements with calculations of the AAA and AXB. For clinical study, 10 patients who had undergone whole pelvic radiotherapy (WPRT) followed by prostatic bed-only radiotherapy (PBRT) using VMAT were enrolled. An ERB was used for PBRT but not for WPRT. To compare dosimetric parameters, the cumulative dose-volume histograms, mean, maximum, and minimum doses were measured for the planning target volume. Homogeneity of plans were confirmed using V95%, V107% (VX%, percentage volumes receiving at least X% of prescribed doses) and conformity indices (homogeneity index [HI], conformity index [CI], and conformation number [CN]). We compared volumes of the organ-at-risk receiving 10% to 100% (10-tier at 10% interval) of prescribed doses (V10% – V100%).ResultsIn the phantom study, the AAA showed larger disagreement with the measurements, and overestimated the dose in the air cavity, comparing with the AXB. For WPRT planning, the AAA predicted a lower maximum dose and V107% than the AXB. For PBRT planning, the AAA estimated a higher minimum dose, lower maximum dose, and smaller V107%, and larger V95% than the AXB. Regarding the conformity indices, the AAA was estimated to be more homogenous than the AXB for PBRT planning (HI, 0.088 vs. 0.120, p = 0.005; CI, 1.052 vs. 1.038, p = 0.022; and CN, 0.920 vs. 0.900, p = 0.007) but not for WPRT planning. Among V10% to V100% of the rectum, the PBRT exhibited significant discrepancies in V30%, V40%, V70%, V80%, and V90%; while the WPRT did in V20% and V30%.ConclusionsThe phantom study demonstrated that the AXB calculates more accurately in the air cavity than the AAA. In the clinical setting, the AXB exhibited different dosimetric distributions in the VMAT plans for PBRT containing an ERB. The AXB should be considered for prostate cancer patients irradiated with an ERB for better applying of heterogeneous condition.Electronic supplementary materialThe online version of this article (doi:10.1186/s13014-015-0346-3) contains supplementary material, which is available to authorized users.