The advancement of Industry 4.0 and Industrial Internet of Things (IIoT) has laid more emphasis on reducing the parameter amount and storage space of the model in addition to the automatic and accurate fault diagnosis. In this case, this paper proposes a lightweight convolutional neural network (LCNN) method for intelligent fault diagnosis of bearing, which can largely satisfy the need of less parameter amount and storage space as well as high accuracy. First, depthiwise separable convolution is adopted, and a LCNN structure is constructed through an inverse residual structure and a linear bottleneck layer operation. Secondly, a novel decomposed Hierarchical Search Space is introduced to automatically explore the optimal LCNN for bearing fault diagnosis in the context of the IIoT. In the meantime, the real-time monitoring and fault diagnosis of the model are also deployed. In order to verify the validity of the designed model, Case Western Reserve University Bearing fault dataset and MFPT bearing fault dataset are adopted. The results demonstrate the great advantages of the model. The LCNN model can automatically learn and select the appropriate features, highly improving the fault diagnosis accuracy. Meanwhile, the computational and storage costs of the model are largely reduced, which contributes to its being applied to the mechanical system in the IIoT context. INDEX TERMS Fault diagnosis, lightweight convolutional neural network, decomposed Hierarchical Search Space, Industry 4.0, the industrial Internet of Things.