There are many disadvantages for traditional Convolutional Neural Network (CNN) in rolling bearing fault diagnosis, such as low efficiency, weak noise immunity, and poor generalization with changing load. To solve the problem, this paper proposes a methodology of improved parallel CNN (IPCNN). In IPCNN, the simple pooling layer is removed and the parallel structure is to stack directly convolutional layers, with three branches, each branch has 4 layers, where the convolution kernels are all 3 × 3 and the stride sizes are 1, 2, and 3, respectively. The structure that is capable of feature fusion can extract features from the input information. Subsequently, the global average pooling (GAP) layer is used for down sampling, and the bearing faults are classified by the fully connected (FC) layer. In addition, the effectiveness of the proposed model structure is verified by testing the datasets. To further verify the validity of the model, the performance of the model was evaluated by diagnostic accuracy, prediction time, SD, and model size. In order to verify the noise immunity and generalization of the proposed model, the AlexNet, Vgg16, and ResNet18 models are compared, respectively. By performing 2D gray images transformation on the Case Western Reserve University (CWRU) bearing data, rolling bearing fault diagnosis method based on IPCNN model has higher efficiency, stronger noise resistance in the noise environment, and better generalization ability when the load changes.