This paper presents a novel plasticity-based formulation for three-dimensional (3D) topology optimization of continuum structures. The proposed formulation addresses the optimization problem by combining mixed rigid-plastic analysis with density-based topology optimization, resulting in a volume minimization approach. Unlike conventional stress-constrained topology optimization methods that rely on linear elastic structural analysis, our developed formulation focuses on enhancing the loading capacity of the designed structures based on the plastic limit theory, leading to more cost-effective designs. To improve computational efficiency, we employ the smoothed finite element technique in our proposed method, enabling the utilization of linear tetrahedral elements for 3D mesh refinement. Moreover, the final formulation of our developed method can be efficiently solved using the advanced primal–dual interior point method, eliminating the need for a separate nonlinear finite element structural analysis. Numerical examples are presented to demonstrate the effectiveness of the proposed approach in offering enhanced design possibilities for continuum structures.