This paper proposed a high-speed target tracking algorithm based on the pulse-sequencebased image sensor. The algorithm separates the target from the background through the statistical distribution of the pulses in a single frame of data. In order to achieve more accurate target tracking, the phenomenon of image lag in the process of capturing the high-speed moving target is studied. The positioning deviation is compensated by estimating the image lag length of the moving target. The algorithm is implemented with pipeline processing on field-programmable gate array (FPGA). The positioning accuracy of the algorithm is verified with input data from a pulse-sequence image sensor model. The verification results show that the maximum position deviation of the target at different speeds is 0.94 pixels. In the experiment of tracking a 40 m/s moving ball, this algorithm can provide real-time position information within 44 ns.