Rail track, turnout and level crossing are critical parts of railway components. So faults of rail surface are directly related to the operation safety. The defects occurring in switches can lead to accidents and the derailment of the train. Therefore, it is necessary to utilize advanced technology to monitor rail damage of the switch and level crossing zones. In this study, the switch and level crossing detection are performed with vision based contactless image processing techniques. Edge detection algorithm, image processing filters, morphological feature extraction, Hough transform and hierarchical SVM classifier are utilized to detect turnout and level crossing regions in the railway images. While switch points are detected, images of the area in front of the train are continuously taken from camera placed on the train. These images contain foreign objects such as buildings, trees, etc. The foreign objects existing in the railway image make it difficult to detect accurately the rail track. The turnout and level crossing detection cannot also be performed using the information of the rail track that cannot be correctly detected. The several processes, such as some image processing filters and morphological feature extraction, have been performed to solve this problem. With destroying the foreign objects from the railway images, both the accuracy rate of the proposed method has been increased and real-time implementation of the method is provided. The Hough transform is utilized to detect rail track, switch and level crossing in the proposed method. The turnout detection is performed by applying SVM classifier to railway image. The proposed method provides real-time rail track, level crossing and turnout detection. Furthermore, this method has gained dynamic feature due to not be utilized any template image for switch detection. When evaluated from these aspects, both the proposed method is superior to other methods in literature and results were obtained in a shorter time thanks to applying ROI Segmentation to the railway image.