Building on our previous work [Phys. Rev. D82, 085016 (2010)], we show in this paper how a Brownian motion on a short scale can originate a relativistic motion on scales that are larger than particle's Compton wavelength. This can be described in terms of polycrystalline vacuum. Viewed in this way, special relativity is not a primitive concept, but rather it statistically emerges when a coarse graining average over distances of order, or longer than the Compton wavelength is taken. By analyzing the robustness of such a special relativity under small variations in the polycrystalline grain-size distribution we naturally arrive at the notion of doubly-special relativistic dynamics. In this way, a previously unsuspected, common statistical origin of the two frameworks is brought to light. Salient issues such as the rôle of gauge fixing in emergent relativity, generalized commutation relations, Hausdorff dimensions of representative path-integral trajectories and a connection with Feynman chessboard model are also discussed.