Under complex seismic forces, the failure characteristics of the plastic hinge region at the bottom of the pier column and the methods improving the ductility have attracted extensive attention. In this study, steel fiber-reinforced concrete with fine aggregate (SFRC-FA) was applied to locally replace the conventional concrete in the potential plastic hinge region at the bottom of the pier column. Five SFRC-FA pier column specimens with different stirrup ratios and different replacement lengths and one conventional reinforced concrete pier column specimen were produced. Using the seismic behavior tests under the combined bending-shear-torsion-axial force, the failure mode, torsional bearing capacity, energy dissipation, and the torsional plastic hinges of the pier columns were investigated. In addition, an equation for calculating the torsional bearing capacity of the new composite pier columns was proposed. The results showed that (1) compared with the reinforced concrete pier column, the plastic hinge was shifted from the bottom of the pier column to the middle of the height of the pier column due to the application of SFRC-FA at the bottom of the pier column, which improved the torsional bearing capacity; (2) the effect of reducing the stirrup ratio of the SFRC-FA replacement region on the torsional bearing capacity, cracking mode, energy dissipation, and ductility was not obvious; (3) the accuracy of the new equation based on the space truss model proposed in this article was verified by comparison with the experiments of this study and other researches.