Camera-enabled unmanned aerial vehicles (UAVs) provide a promising technique to considerably speed up the inspection and visual data collection from regions that may otherwise be inaccessible. In addition, the technology of image-based 3D reconstruction can generate a point cloud model using images captured by UAVs. However, the performance of the point cloud modeling may be affected by multiple factors, such as the modeling software, ground control points (GCPs), and UAV flight modes. In this study, three common software packages were compared, and Pix4Dmapper was considered a suitable software for point cloud modeling for earthquake-damaged buildings. The accuracy and resolution of point cloud models are usually evaluated by root mean square error (RMSE) and ground sampling distance (GSD). The effects of the main factors, including the number of GCPs, distribution of GCPs, flight manner of the UAV, and distance from the UAV to the target, were investigated on the basis of two real-world multistory earthquake-damaged structures. The influence rules of the main factors revealed that a close range, automatic flight mode of the UAV, a large number of GCPs, and a relatively wide distribution of the GCPs may generate a point cloud model with low computational costs, high accuracy, and high resolution. In the particular illustration example here, the RMSE is 6.78 mm while the GSD is 1.60 mm. Finally, rapid structural damage inspection was demonstrated using an accurate point cloud model and compared with the inspection results of a total station and terrestrial laser scanner point cloud models. The comparison of different inspection results showed that the relative errors were relatively acceptable within 4%.