This article addresses the problem of distributed average tracking for second‐order multi‐agent systems with disturbances. The unknown disturbances inject into the velocity dynamics, resulting in the degradation of system stability. An extended state observer is designed to estimate the velocities and disturbances. Then, an observer‐based average tracking protocol is proposed, which does not require velocity measurements, and the unknown disturbances are compensated by the estimations. The protocol is implemented in a distributed fashion, without requiring any global information, and is robust to initialization errors. Through Lyapunov analysis, it is verified that the second‐order multi‐agent system can achieve distributed average tracking under the proposed protocol. Finally, the effectiveness of the control method is demonstrated by simulations.