Branching Brownian motion describes a system of particles that diffuse in space and split into offspring according to a certain random mechanism. By virtue of the groundbreaking work by M. Bramson on the convergence of solutions of the Fisher-KPP equation to traveling waves, the law of the rightmost particle in the limit of large times is rather well understood. In this work, we address the full statistics of the extremal particles (first-, second-, third-largest, etc.). In particular, we prove that in the large t-limit, such particles descend with overwhelming probability from ancestors having split either within a distance of order 1 from time 0, or within a distance of order 1 from time t. The approach relies on characterizing, up to a certain level of precision, the paths of the extremal particles. As a byproduct, a heuristic picture of branching Brownian motion "at the edge" emerges, which sheds light on the still unknown limiting extremal process.