The collective dynamics and structure of animal groups has attracted the attention of scientists across a broad range of fields. A variety of agent-based models have been developed to help understand the emergence of coordinated collective behavior from simple interaction rules. A common, simplifying assumption of such collective movement models, is that individual agents move with a constant speed. In this work we critically re-asses this assumption. First, we discuss experimental data showcasing the omnipresent speed variability observed in different species of live fish and artificial agents (RoboFish). Based on theoretical considerations accounting for inertia and rotational friction, we derive a functional dependence of the turning response of individuals on their instantaneous speed, which is confirmed by experimental data. We then investigate the interplay of variable speed and speed-dependent turning on self-organized collective behavior by implementing an agent-based model which accounts for both these effects. We show that, besides the average speed of individuals, the variability in individual speed can have a dramatic impact on the emergent collective dynamics: a group which differs to another only in a lower speed variability of its individuals (groups being identical in all other behavioral parameters), can be in the polarized state while the other group is disordered. We find that the local coupling between group polarization and individual speed is strongest at the order-disorder transition, and that, in contrast to fixed speed models, the group’s spatial extent does not have a maximum at the transition. Furthermore, we demonstrate a decrease in polarization with group size for groups of individuals with variable speed, and a sudden decrease in mean individual speed at a critical group size (N = 4 for Voronoi interactions) linked to a topological transition from an all-to-all to a distributed spatial interaction network. Overall, our work highlights the importance to account for fundamental kinematic constraints in general, and variable speed in particular, when modeling self-organized collective dynamics.