Continuously and reliably estimating the relative configuration of robotic swarms in real-time constitutes a core functionality when pursuing the autonomy of such a swarm. Relying on external positioning systems, such as GPS or motion tracking systems, can provide the required information, but significantly limits the generality of an approach. In this letter, we target formation estimation for autonomous flights of swarms of small UAVs, as they pose particularly challenging restrictions on onboard resources, while opening up a large variety of practical scenarios for a multi-robot setup. While state of the art has been addressing efficient formation estimation, scalability remains limited to only very few agents that can be handled in real-time, with the workload of each agent depending on the total number of agents in the swarm. Aiming for scalable multi-robot systems, here we propose a distributed formation estimation approach, where the computational load of each agent is decoupled from the swarm size. This approach is implemented in a setup with minimal communication effort, requiring only ego-motion estimates from each agent and pairwise distance measurements between them, constraining their configuration globally. Evaluations on swarms of up to 49 UAVs demonstrate the power of our approach to handle large swarms, while keeping the computational load bounded for individual agents and requiring only little data exchange between two robots.