Automobile surface defects, such as scratches and dents, can largely affect the first impressions of consumers. These defects are likely to occur during the processes of manufacturing and painting. Most global automobile companies still rely on visual inspection to detect the defects, which results in instability and inefficiency in the inspection procedure. In this paper, a low‐cost detection system is proposed. The hardware part of the system consists of only light emitting diode (LED) tubes and a single camera. The software part of the system first obtains defect candidates greedily by thresholding binarization result. Then, candidates in each frame are filtered using a differential image. Finally, as the main contribution, our algorithm further filters the candidates using tracking trajectories of defect candidates over multiple frames. Experimental results demonstrate that our method is able to detect defects while suppressing false‐positive detections and is effective in real‐world situations. © 2020 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.