Large-scale dimensional metrology usually requires a combination of multiple measurement systems, such as laser tracking, total station, laser scanning, coordinate measuring arm and video photogrammetry, etc. Often, the results from different measurement systems must be combined to provide useful results. The coordinate transformation is used to unify coordinate frames in combination; however, coordinate transformation uncertainties directly affect the accuracy of the final measurement results. In this paper, a novel method is proposed for improving the accuracy of coordinate transformation, combining the advantages of the best-fit least-square and radial basis function (RBF) neural networks. First of all, the configuration of coordinate transformation is introduced and a transformation matrix containing seven variables is obtained. Second, the 3D uncertainty of the transformation model and the residual error variable vector are established based on the best-fit least-square. Finally, in order to optimize the uncertainty of the developed seven-variable transformation model, we used the RBF neural network to identify the uncertainty of the dynamic, and unstructured, owing to its great ability to approximate any nonlinear function to the designed accuracy. Intensive experimental studies were conducted to check the validity of the theoretical results. The results show that the mean error of coordinate transformation decreased from 0.078 mm to 0.054 mm after using this method in contrast with the GUM method.