Rail inspection is an essential task in railway maintenance and is periodically needed. Inspection is manually operated by trained human operator walking along the track searching for visual anomalies. This monitoring is unacceptable for slowness and lack of objectivity. This paper deals with a patented Visual Inspection System for Railway maintenance, devoted to different tasks. Here, its Rail Detection & Tracking Block (RD&TB) is presented.RD&TB detects and tracks, into the acquired video sequence the rail head, by this way, notably reducing the area to be analyzed and inspected by other modules of VISyR. Thanks to its hardware implementation, RD&TB performs its task in 5.71 μs with an accuracy of 98.5%, allowing an on-the-fly analysis of a video sequence acquired up at 190 km/h. RD&TB is highly flexible and configurable, since it is based on classifiers that can be easily reconfigured in function of different type of rails.