This study explores a visual inspection system incorporating defect detection and the judgment of the defect occurrence side in touch panel fabrication. The surface of a touch panel is transparent glass with conductive lines inside to achieve the purpose of touch control. When touch panels are manufactured, it is the front sides of the products that are primarily processed and are treated with stricter standards. Therefore, it is important to correctly distinguish the defect locations occurring on the front or back side to ensure the quality of the touch panels. We first apply the Hilbert-Huang transform to enhance the contrast between the defects and the background. Second, statistical interval estimation is used to segment background and defects to achieve defect detection. Third, the detected defects on both sides of the substrates are combined and numbered for feature extraction. Fourth, the random forest model is applied to judge where the defects occur on the front or back of the transparent substrates. Experimental results show that the defect detection rate achieves 85.75%, and the false alarm rate is lower to 0.33%. The correct location judgment rate of the defect occurrence is 98.62% on the front and back sides of the touch panels.INDEX TERMS Visual inspection, defect identification, occurrence side judgment, touch panels, Hilbert-Huang transform, random forest model.