Due to the lack of data available for training, deep learning hardly performed well in the field of garbage image classification. We choose the TrashNet data set which is widely used in the field of garbage image classification, and try to overcome data deficiencies in this field by optimizing the network structure. In this paper, it is found that the deeper network and short-circuit connection, which are generally accepted in the field of deep learning, will not work well on the TrashNet data set. By analyzing and modifying the network structure, we propose an effective method to improve the network performance on TrashNet data set. This method widens the network by expanding branches, and then uses add layers to realize the fusion of feature information. It can make full use of feature information at slight additional computational cost. Using this method to replace the core structure of the Xception network, the performance of the improved network has been improved greatly. Finally, the M-b Xception network proposed by us achieves 94.34% classification accuracy on the TrashNet data set, and has certain advantages over some state-of-theart methods on multiple indicators. The python code can be download from https://github.com/scp19801980/Trash-classify-M_b-Xception.