Breast cancer is a global threat to women’s health. Three-dimensional (3D) automated breast ultrasound (ABUS) offers reproducible high-resolution imaging for breast cancer diagnosis. However, 3D-input deep networks are challenged by high time costs, a lack of sufficient training samples, and the complexity of hyper-parameter optimization. For efficient ABUS tumor classification, this study explores 2D-input networks, and soft voting (SV) is proposed as a post-processing step to enhance diagnosis effectiveness. Specifically, based on the preliminary predictions made by a 2D-input network, SV employs voxel-based weighting, and hard voting (HV) utilizes slice-based weighting. Experimental results on 100 ABUS cases show a substantial improvement in classification performance. The diagnosis metric values are increased from ResNet34 (accuracy, 0.865; sensitivity, 0.942; specificity, 0.757; area under the curve (AUC), 0.936) to ResNet34 + HV (accuracy, 0.907; sensitivity, 0.990; specificity, 0.864; AUC, 0.907) and to ResNet34 + SV (accuracy, 0.986; sensitivity, 0.990; specificity, 0.963; AUC, 0.986). Notably, ResNet34 + SV achieves the state-of-the-art result on the database. The proposed SV strategy enhances ABUS tumor classification with minimal computational overhead, while its integration with 2D-input networks to improve prediction performance of other 3D object recognition tasks requires further investigation.