Visual SLAM (Simultaneous Localization And Mapping) is a technique mainly used for robot navigation and positioning. It uses vision sensors to map the environment and estimate its own location. In dynamic environment, they are inevitably affected by dynamic objects, which leads to the decrease of accuracy and stability of the system. In this paper, the visual odometer in SLAM system is studied, and the semi-direct visual odometer (SVO) algorithm is improved on the basis of ORB-SLAM framework. The basic principle of the algorithm is to treat pixel matching between adjacent image frames as an optimization problem, and use direct registration method to obtain camera pose for sparse feature blocks, avoiding a lot of feature extraction and matching. When a key frame appears, ORB (Oriented FAST and Rotated BRIEF) features of the image are extracted, and feature matching is used to track local maps to obtain more mapping relationships. Moreover, camera pose and three-dimensional structure of the scene are estimated by minimizing reprojection errors, thus improving positioning accuracy and map construction quality. Finally, experiments on several public data sets show that the improved semi-direct visual odometer scheme in this paper still has good robustness to moving objects in dynamic scenes. While maintaining high accuracy, it obtains better real-time performance by tracking visual feature points.