Traditional inspection of bridge surfaces is often inefficient and requires inspectors to inspect in high-risk areas. For the time being, Unmanned Aerial Vehicles (UAVs), as an efficient and safe means of inspection, can be equipped with high-definition cameras, and are utilised for data collection in unmanned inspections, especially in areas that are difficult for people to reach. However, how to manage the crack data obtained from UAV and machine learning recognition is still a problem. In addition, bridge modelling (BIM) based on geometric and semantic information can be applied to the inspection of bridge surfaces. To this end, this paper proposes a method for detecting and modelling bridge defects based on UAV and BIM, that is, a method for managing bridge defects by automatically identifying and locating the defect data by combining the images acquired by the UAV, which can be combined with machine vision techniques, mapping and modelling the defect data to BIM, and modelling defects from BIM as objects. Firstly, the bridge defect images captured by UAV are processed and some useful data such as coordinates are extracted from them. In this paper, a simplified coordinate method is proposed to convert the locations of the defects existing in the actual project into the coordinates in the BIM model. Meanwhile, on this basis, this paper utilises machine vision-based bridge crack detection, which is used to detect defects in the captured images and perform feature extraction on them. Finally, by modelling the identified defects, a new object with detailed information is obtained and mapped to the corresponding location in the BIM. The effectiveness of this approach is demonstrated by analysing the example of the Martyrs River Bridge. The study will be applied to combine the defects of the bridge with the BIM model, which will combine the existing state and the data from the BIM in order to perform structural inspections during the repair process.