Railways play a vital role in the inland transportation system worldwide, and abnormal bolt components at the track joints are the main cause of train accidents. The detection and identification of faults in rail bolt components are of considerable research importance. To address this problem, numerous researchers have opted for computer vision-based methods to accomplish the detection and identification of the target, but the existing methods have poor detection performance diminished detection capabilities when the target position changes or some feature information is occluded, and the detection speed and accuracy are far from meeting the requirements of practical applications. Therefore, based on the construction of a dedicated dataset for bolt components, this paper uses the K-means dimensional clustering algorithm to re-cluster the dataset according to the target size characteristics, with the aim of reduce the bounding box regression error. At the same time, a novel loss function iteration method is proposed by incorporating an adaptive optimization algorithm, in order to improve the detection speed and ensure good convergence, and the model complexity is reduced based on deep model pruning. Finally, the optimized detection model is implemented on the robotic-assisted platform for testing, and the experimental results verify that the algorithm can quickly and accurately complete various fault diagnosis tasks of bolt components in practical applications. The main achievements of this study include the construction of a large-scale image dataset for novel rail bolt components and propelled the application of deep learning methods in vision-based rail bolt fault diagnosis.