To achieve photogrammetry without ground control points (GCPs), the precise measurement of the exterior orientation elements for the remote sensing camera is particularly important. Currently, the satellites are equipped with a GPS receiver, so that the accuracy of the line elements of the exterior orientation elements could reach centimeter-level. Furthermore, the high-precision angle elements of the exterior orientation elements could be obtained through a star camera which provides the direction reference in the inertial coordinate system and star images. Due to the stress release during the launch and the changes of the thermal environment, the installation matrix is variable and needs to be recalibrated. Hence, we estimate the cosine angle vector invariance of a remote sensing camera and star camera which are independent of attitude, and then we deal with long-term on-orbit data by using batch processing to realize the accurate calibration of the installation matrix. This method not only removes the coupling of attitude and installation matrix, but also reduces the conversion error of multiple coordinate systems. Finally, the geo-positioning accuracy in planimetry is remarkably higher than the conventional method in the simulation results.