In this paper, an angle-of-arrival (AOA)-based algorithm is proposed for tracking the position of an anonymous target in three-dimensional (3D) space. Distributed sensors are deployed, which can measure both the azimuth and elevation angles of the AOAs. Assuming the target movement is non-linear, the extended Kalman filter (EKF) is applied, where the observation process is realized by a practical AOA-based position detector, to form a unified factor graph (FG) framework. Moreover, the variance of observation errors, which is needed by EKF, is estimated in real time by using both the AOA measurements and the predicted target state. Such a dynamic estimating approach exhibits higher performance robustness compared to the conventional method, especially when the sensing environment is unstable. Additionally, the predicted target state is also used as the a priori information of the system, in order to reduce the impacts of burst sensing errors. According to the simulations, the proposed system is shown to achieve less root mean squared errors (RMSE) in different evaluation scenarios, with fast convergence behavior.