Defect detection of workpiece surface is a basic and essential task in the production of workpiece products. Although significant progress has been made in workpiece surface defect detection, traditional methods are still difficult to detect small defects efficiently. To deal with this problem, an efficient small defect detection network (ESDDNet) with novel parallel convolution module, serial convolution module and feature fusion module is proposed. Firstly, lightweight backbone network is used to extract preliminary defect features. Secondly, the parallel convolution module and serial convolution module are used to obtain abundant defect features. Then, the feature fusion module is used to fuse shallow features with deep features to enhance the features of small defects. Finally, the obtained features are put into the corresponding detection head to get the final prediction results. The experimental results on local cable dataset and public PCB dataset show that our method achieves remarkable performance in detecting small defects and achieves favorable trade-off between accuracy, speed and model size, which meets the requirements of industrial application.