In the present study, an automatic defect detection system has been assembled and introduced for Polyvinyl chloride (PVC) powder. The average diameter for PVC powder is approximately 100 μm. The system hardware includes a powder delivery device, a sieving device, a circular platform, an image capture device, and a recycling device. A defect detection algorithm based on YOLOv4 was developed using CSPDarkNet53 as the backbone for feature extraction, spatial pyramid pooling (SPP) and path aggregation network (PAN) as the neck, and Yoloblock as the head. An auto-annotation algorithm was developed based on a digital image processing algorithm to save time in feature engineering. Several hyper-parameters have been employed to improve the efficiency of detection in the process of training YOLOv4. The Taguchi method was utilized to optimize the performance of detection, in which the mean average precision (mAP) is the response. Results show that our optimized YOLOv4 has a test mAP of 0.9385, compared to 0.8653 and 0.7999 for naïve YOLOv4 and Faster RCNN, respectively. Additionally, with the optimized YOLOv4, there is no false alarm for images without any foreign matter.