Convolutional neural networks have been widely used in nondestructive testing of agricultural products. Aiming at the problems of missing detection, false detection, and slow detection, a lightweight improved cottonseed damage detection method based on YOLOv5s is proposed. Firstly, the focus element of the YOLOv5s backbone network is replaced by Denseblock, simplifying the number of modules in the backbone network layer, reducing redundant information, and improving the feature extraction ability of the network. Secondly, the collaborative attention (CA) mechanism module is added after the SPP pooling layer, and a large target detection layer is reduced to guide the network to pay more attention to the location, channel, and dimension information of small targets. Thirdly, Ghostconv is used instead of the conventional convolution layer in the neck feature fusion layer to reduce the amount of floating-point calculation and speed up the reasoning speed of the model. The CIOU loss function is selected as the border regression loss function to improve the recall rate of the model. Lastly, the model was verified using an ablation experiment and compared with the YOLOv4, Yolov5s, and SSD-VGG16 network models. The accuracy, recall rate, and map value of the improved network model were 92.4%, 91.7%, and 98.1%, respectively, and the average recognition time of each image was 97 fps. The results show that the improved network can effectively solve the problem of missing detection, reduce false detection, and have better recognition performance. This method can provide technical support for real-time and accurate detection of damaged cottonseed in a cottonseed screening device.