This paper is concerned with observer-based trajectory control of directional drilling system with rotary steering devices. Since the trajectory orientation cannot be measured directly, most of the existing results implicitly assume that the trajectory orientation is equal to the measurement of orientation of the bottom hole assembly (BHA) or the drill bit, which would bring in certain control errors in practice. To avoid such control errors, in this paper, a trajectory control approach is established, which includes a state observer to estimate the trajectory orientation based on the measurement of the BHA orientation. Firstly, the relationship between the trajectory orientation and the BHA orientation is analyzed, and a trajectory evolution model is introduced according to the analysis. Secondly, based on this trajectory evolution model, a variable transformation technique is applied such that a trajectory tracking control problem is formulated. Thirdly, a feedback control strategy is devised based on the estimation of the trajectory orientation for trajectory tracking. Fourthly, stability analysis of the closed-loop system is discussed, and the design of the controller and state observer is done.