In fault diagnosis, broad learning systems (BLS) have been applied in recent years. However, the best fault diagnosis cannot be guaranteed by width node extension alone, so a stacked broad learning system (stacked BLS) was proposed. Most of the methods for choosing the number of depth layers used optimization algorithms that tend to increase computation time. In addition, the data under single feature selection are not sufficiently representative, and effective features are easily lost. To solve these problems, this article proposes an infrared fault diagnosis model for rolling bearings based on integration of principal component analysis and singular value decomposition (IPS) and the stacked BLS with self-selected depth model (SSDStacked-BLS). First, 72 second-order statistical features are extracted from the pre-processed infrared images of rolling bearings. Next, feature selection is performed using IPS. he IPS feature selection module consists of principal component analysis (PCA) and singular value decomposition (SVD). The feature selection is performed by PCA and SVD separately, which are then stitched together to form a new feature. This ensures a comprehensive coverage of infrared image features. Finally, the acquired features are input into SSDStacked-BLS. This model establishes a data storage group for the residual training characteristics of stacked BLS, adding one block at a time. The accuracy rate of each newly added block is output and saved to the data storage group. If the diagnostic rate fails to increase three consecutive times, the block stacking is stopped and the results are output. IPS-SSDStacked-BLS achieved an accuracy of 0.9667 in 0.1775 s. This is almost five times faster than stacked BLS optimized using the grid search method. Compared with the original BLS, its accuracy was 0.0445 higher and the time was approximated. Compared with IPS-SVM, IPS-RF, IPS-1DCNN and 2DCNN, IPS-SSDStacked-BLS was more advantageous in terms of accuracy and time consumption.