Several studies have shown the influence of the relative streaming velocity (SV) between baryons and dark matter on the formation of structures. For the first time, we constrain the local value of the SV in which the Milky Way was formed. We use the semi-analytical model a-sloth to simulate the formation of Milky Way-like galaxies. The high resolution in mass and time of the dark matter merger trees from the Caterpillar simulation enables us to accurately model star formation in the smallest progenitor halos at high redshift. The efficient semi-analytical nature of a-sloth allows us to run many simulations with various values of the local SV. Our investigation on the influence of the SV shows that it delays star formation at high redshift. However, at redshift z=0, the SV has no effect on the total stellar mass in the Milky Way nor its Satellites. We find that extremely metal-poor and ultra metal-poor stars are affected by the SV, and can hence be used to constrain its local value. The local optimal value of the SV is $v_\mathrm{SV} =1.75^{+0.13} _{-0.28}\, \sigma _\mathrm{SV}$, which is based on four independent observables. We further find that the SV decreases the number of luminous Milky Way satellites, but this decrease is not enough to solve the missing satellite problem.