Railway alignment optimization is considered one of the most complicated and time‐consuming problems in railway planning and design. It requires searching among the infinite potential alternatives in huge three‐dimensional (3D) search spaces for a near‐optimal alignment, while considering complex constraints and a nonlinear objective function. In mountainous regions, the complex terrain and constructions require additional and more complex constraints than in topographically simpler regions. In this paper, the authors solve this problem with an algorithm based on a 3D distance transform (3D‐DT). Compared with previous two‐dimensional distance transform (2D‐DT) methods developed in this field, the feasible search spaces of 3D‐DT are greatly increased. Consequently, this new method can find more alternatives with higher qualities. In this approach, an erythrocyte‐shaped 3D neighboring mask is developed to narrow local search spaces and speed up the search process. Besides, a stepwise‐backstepping strategy is designed to dynamically determine feasible 3D search spaces and efficiently search the study area. During the 3D‐DT search process, multiple constraints, including geometric, construction, and location constraints, are effectively handled. After the 3D‐DT search, a genetic algorithm is employed to optimize the 3D‐DT paths into final alignments. Finally, this novel approach is applied to an actual case in a complex mountainous region. The comprehensive cost of the best solution generated by 3D‐DT is 16% below a manual solution produced by very experienced human designers. Furthermore, the total number of feasible alternatives found by 3D‐DT is 4.3 times greater than by 2D‐DT. The comprehensive cost of the best 3D‐DT solution is 10% below the best one generated by 2D‐DT.