The transmission of power and motion in road vehicles with internal combustion engines is achieved by different design variants of differential transmissions. The open differential transmission (ODT) is installed to a greater extent in passenger cars with rear-wheel drive due to its simpler design. Due to its robustness, it is possible to reduce its mass. Reducing the mass of the ODT, as well as reducing the mass of the other design elements of the vehicle, contributes to reducing the overall mass of the vehicle and improves the energy efficiency of the vehicle. The paper develops and proposes an algorithm that combines the design of the ODT according to ISO 23509:2006, the numerical calculation of the design elements (ring gear and drive shaft with pinion) using the finite element method (FEM) and the numerical global–local model, the topological optimization method (TOM) and the results of the FEM analysis in determining the design parameters. In addition, the proposed algorithm uses the application of the response surface method (RSM) in the construction of a mathematical model. With the proposed mathematical model, the mathematical objective function of the ODT overall mass reduction describes the influence of the previously selected design parameters on the overall mass reduction of the ODT. The mathematical model is also used to analyze the partial influence of the design parameters on the objective functions of the partial mass reduction of the ring gear and pinion drive shaft. Using the R2 and root mean square error (RMSE), an accuracy check of the proposed mathematical model was performed. According to the proposed algorithm and mathematical model, the two mentioned design elements of the ODT were optimized. After optimization, the overall mass of the ODM was reduced by 16.5%.