Train incidents with animals and even humans have gotten more attention in the past. Every year, many animals are killed on train tracks, causing an imbalance of the ecosystem and significant delays in railway traffic. Similarly, sometimes train accident is prevalent at the rail gates, where motor vehicles are crossing the train line. All that happens due to the lack of detecting the objects correctly on the train line. Since the detection depends on the driver or human, there is a possibility of occasionally making an error in honking the horn at the right moment, leading to the accident. That's why there needs to be an automated solution to detect the objects on the train line and inform the driver about them. In this research, we proposed a system that detects objects only on the train line. Sometimes, there is an object just beside the train line, which is at a safe distance. To eliminate this kind of wrong detection, we detected the train line first and then detected the objects on the detected train line. Here, we used the Mask R-CNN algorithm to detect the train line and things on the train line. A railway traffic dataset with an input size of 512 x 512 pixels was used to test the proposed methodology, which came up with results with a mean average precision of 0.9375 and a frame rate of 30 frames per second. According to the findings of the experiments, the suggested approach can apply to identify objects on railroads in the real world.