In real-world fault diagnosis, the process of acquiring fault vibration signals frequently faces various challenges, which lead to certain fault types being represented by a scarcity of vibration signal data. Although support matrix machine (SMM) exhibits excellent classification ability in matrix-based data, its ideal model is difficult to establish with limited labeled samples. To address this limitation, a novel angle-based transfer support matrix machine (ATSMM) is proposed, which integrates matrix-based classifier and transfer learning theory. ATSMM employs an angle-based transfer learning method, which aims to reduce the angle between the normals of the target domain hyperplane and the source domain hyperplane, enabling the transfer of structural information across different domains. Meantime, ATSMM effectively addresses the issue of negative transfer that frequently occurs in transfer learning, which adds the weights calculated by the maximum mean discrepancy method to further improve the transfer effectiveness of the target domain model. The superiority of ATSMM is verified by two different roller bearing datasets, and the outcomes of experiments exhibit that the proposed ATSMM has excellent classification performance compared to alternative methods, especially in scenarios where the labeled sample size is limited.