Accurate theoretical models of actuators are crucial for researching autonomous driving in heavy-duty commercial vehicles (HDCVs). A hydraulic retarder is a widely used auxiliary braking device in HDCVs that prevents thermal failure of the main brake resulting from continuous braking. The working chamber and the control system are closely interconnected within the hydraulic retarder, forming a highly nonlinear system. Therefore, most mathematical models for predicting the braking torque of hydraulic retarders have separated the working chamber from the control systems, making it difficult to explain the differences between the hydraulic retarder's predicted and actual performance. This paper establishes a novel mathematical model incorporating the working chamber and control system analysis for a hydraulic retarder based on Rankine vortex dynamics. A functional relationship was obtained between the hydraulic retarder's braking torque and the rotor speed, control pressure, float chamber pressure, oil density, and characteristic parameters of the working chamber. The mathematical model has been verified through computational fluid dynamics (CFD) and experiments. CFD results show the velocity distribution of oil vortex flow in the working chamber, and the variation laws of the fundamental parameters are consistent with the established mathematical model. The average errors between the mathematical model calculated and experimental braking torque are 6.3%, 9.5%, 11.8%, and 4.2% at control pressures of 2.8, 2.2, 1.4, and 0.6 bar, respectively, confirming the mathematical model's effectiveness. The mathematical model holds significant value for the design and development of hydraulic retarders and the control strategies of HDCVs with hydraulic retarders.