In response to the growing demand for railway obstacle monitoring, lidar technology has emerged as an up-and-coming solution. In this study, we developed a mechanical 3D lidar system and meticulously calibrated the point cloud transformation to monitor specific areas precisely. Based on this foundation, we have devised a novel set of algorithms for obstacle detection within point clouds. These algorithms encompass three key steps: (a) the segmentation of ground point clouds and extraction of track point clouds using our RS-Lo-RANSAC (region select Lo-RANSAC) algorithm; (b) the registration of the BP (background point cloud) and FP (foreground point cloud) via an improved Robust ICP algorithm; and (c) obstacle recognition based on the VFOR (voxel-based feature obstacle recognition) algorithm from the fused point clouds. This set of algorithms has demonstrated robustness and operational efficiency in our experiments on a dataset obtained from an experimental field. Notably, it enables monitoring obstacles with dimensions of 15 cm × 15 cm × 15 cm. Overall, our study showcases the immense potential of lidar technology in railway obstacle monitoring, presenting a promising solution to enhance safety in this field.