The spot scanning defect detection system has good detection performance for wafer defects. However, due to the existence of system assembly errors, the wafer defect inspection system could not scan following the planned trace, which resulted in distortion of the reconstructed image and had a greater impact on defect feature extraction and accurate defect location. The system deviation calibration and adjustment can further facilitate the accuracy of the system position and image reconstruction. To address this issue, the causes and shapes of imaging distortion were analyzed and simulated, and a simple and effective calibration method based on a scanning deviation model for the spot scanning defect detection system was proposed. First, a scanning deviation model was established following the characteristics of the spot scanning system, and different degrees of reconstructed image distortion were simulated and analyzed as the system deviation changes. Then, an 8-inch wafer with a grid was used as the calibrator and scanned using the developed spot scanning system. Serveral grid corners in the reconstructed image are selected and used as the calibration feature points. The constraint function is constructed using the equidistant features between adjacent grid points, and then the quasi-Newton method is used to optimize the parameters of the function to obtain the best deviation estimation of the spot scanning system. To prove the validity of the estimated deviation parameters, another patterned wafer was scanned and the distortion image was used for reconstruction. The experiment demonstrated that an accurate and distortion-free wafer image can be reconstructed using the calibrated deviation parameters, and the root mean square error of the measured distance between two points of interest was only 0.61 pixels, which proved the effectiveness of the proposed calibration method.