Due to advances in semiconductor processing technologies, each slice of a semiconductor is becoming denser and more complex, which can increase the number of surface defects. These defects should be caught early and correctly classified in order help identify the causes of these defects in the process and eventually help to improve the yield. In today’s semiconductor industry, visible surface defects are still being inspected manually, which may result in erroneous classification when the inspectors become tired or lose objectivity. This paper presents a vision-based machine-learning-based method to classify visible surface defects on semiconductor wafers. The proposed method uses deep learning convolutional neural networks to identify and classify four types of surface defects: center, local, random, and scrape. Experiments were performed to determine its accuracy. The experimental results showed that this method alone, without additional refinement, could reach a top accuracy in the range of 98% to 99%. Its performance in wafer-defect classification shows superior performance compared to other machine-learning methods investigated in the experiments.