The magneto-resistance of antidot lattices shows pronounced peaks, which became a hallmark of ballistic electron transport. While most studies agree that they reflect the interplay of regular and chaotic motion in the quasi-classical dynamics, the exact mechanism has been surprisingly controversial. Inspired by recent experiments on graphene antidot lattices showing that the effect survives strong impurity scattering, we give a new explanation of the peaks linked to a fundamental relation between collision times and accessible phase space volumes, accounting for their robustness. Due to the fundamental nature of the mechanism described it will be relevant in many mesoscopic transport phenomena.