Automatic detection of anomalies on the metal surface is an essential capability in industries to provide the better-quality control. To locate and identify the type of defect, it is necessary to find the Region of interest (RoI) from the captured image. Segmentation of the captured image is one among the many methods to achieve this task. Therefore, a precise and accurate segmentation method has major role to improve the metal surface anomaly detection rate in industry. As the defects are different in it’s size, shape and type, the process of semantic segmentation for metal surface is considered as a challenging task. To address this issue, a deep learning based high calibre U- shaped network is proposed. It can be considered as an automatic quality control system for industries. The proposed method is effective in predicting the presence of defects. The system is also capable to locate the position of the defect on surface without the intervention of human being. The up-sampling technique provided with the convolutional neural network in the architecture makes the system to produce high resolution outputs. The proposed system has been evaluated based on accuracy, precision, loss and IoU after training and testing the model using two different datasets called NEU metal surface defect database and Kolektor surface defect data set.