In industrial production, detecting polarizer defects online and in real time is necessary. Existing methods of detecting polarizer defects based on deep learning can ensure the accuracy of classification; however, there are several issues associated with these methods. These include the models having low detection speed, consuming large amounts of memory, and being difficult to be transplanted into online detection systems. To solve the aforementioned problems, a lightweight efficient network (LWEN) structure based on deep learning was designed, which improves the standard convolution layer and the fully connected (FC) layer to minimize the training model size and increase the speed of classification without reducing the accuracy of classification. First, a new building block, the shunt module, was designed to build the LWEN.Subsequently, a global average pooling layer was used to reduce the spatial resolution to 1 before the FC layer. These key technologies were designed to reduce the number of network parameters and minimize the model size of the network. Experimental results show that the proposed LWEN outperforms the state-of-the-art approaches in terms of classification accuracy, speed, and model size.