Counting the number of passengers getting in/out of a bus in each station is a challenging task in smart bus field. To solve the problem, this paper presents a method of automatic people counting for a bus based on a RGB-D video, a zenithal camera in the bus door capturing the passengers flow is set, and the camera can get depth Image and RGB image simultaneously, we proposed a method combining RGB image and depth image to detect the head of the passengers, and then a novel tracking strategy is proposed. The test results show that this method can accurately extract the head area in the video and can accurately count the number of people getting in/out of a bus at a fast speed. The test data in different time periods show that the method has strong robustness and stability.