With the rapid development of light-rail public transportation, video-based obstacle detection is becoming an essential and foregoing task in driver assistance systems. The system should be able to automatically survey the tramway using an onboard camera. However, the functioning of the system is challenging due to the presence of various ground types, different weather and illumination conditions, as well as varying time of acquisition. This article presents a real-time tramway detection method that deals efficiently with various challenging situations in real-world urban rail traffic scenarios. It first uses an adaptive multilevel thresholding method to segment the regions of interest of the tramway, in which the threshold parameters are estimated using a local accumulated histogram. The approach then adopts the region growing method to decrease the influence of environmental noise and to predict the trend of the tramway. The experiment validation of this study proves that the method is able to correctly detect tramways even in challenging scenarios and uses lesser computational time to meet the real-time demand.