The development of intelligent manufacturing often focuses on production flexibility, customization and quality control, which are crucial for chip manufacturing. Specifically, defect detection and classification are important for manufacturing processes in the semiconductor and electronics industries. The intelligent detection methods of chip defects are still challenge and have always been a particular concern of chip processing manufactures in an automated industrial production line. YOLOv4 method has been widely used for object detection due to its accuracy and speed. However, there are still difficulties and challenges in the detection for small targets, especially defects on chip surface. This study proposed a small object detection method based on YOLOv4 for small object in order to improve the performance of detection. It includes expanding feature fusion of shallow features; using k-means++ clustering to optimize the number and size of anchor box; and removing redundant YOLO head network branches to increase detection efficiency. The results of experiments reflect that SO-YOLO is superior to the original YOLOv4, YOLOv5s, and YOLOv5l models in terms of the number of parameters, classification and detection accuracy.