In this study, to solve the low accuracy of multi-space-based sensor joint tracking in the presence of unknown noise characteristics, an adaptive multi-sensor joint tracking algorithm (AMSJTA) is proposed. First, the coordinate transformation from the target object to the optical sensors is considered, and the observation vector-based measurement model is established. Then, the measurement noise characteristics are assumed to be white Gaussian noise, and the measurement covariance matrix is set as a constant. On this premise, the traditional iterative extended Kalman filter is applied to solve this problem. However, in most actual engineering applications, the measurement noise characteristics are unknown. Thus, a forgetting factor is introduced to adaptively estimate the unknown measurement noise characteristics, and the AMSJTA is designed to improve the tracking accuracy. Furthermore, the lower bound of the proposed algorithm is theoretically proved. Finally, numerical simulations are executed to verify the effectiveness and superiority of the proposed AMSJTA.