Ultrasonic time-of-flight (ToF) measurements enable the non-destructive characterization of material parameters as well as the reconstruction of scatterers inside a specimen. The time-consuming and potentially damaging procedure of applying a liquid couplant between specimen and transducer can be avoided by using air-coupled ultrasound. However, to obtain accurate ToF results, the waveform and travel time of the acoustic signal through the air, which are influenced by the ambient conditions, need to be considered. The placement of microphones as signal receivers is restricted to locations where they do not affect the sound field. This study presents a novel method for in-air ranging and ToF determination that is non-invasive and robust to changing ambient conditions or waveform variations. The in-air travel time was determined by utilizing the azimuthal directivity of a laser Doppler vibrometer operated in refracto-vibrometry (RV) mode. The time of entry of the acoustic signal was determined using the autocorrelation of the RV signal. The same signal was further used as a reference for determining the ToF through the specimen in transmission mode via cross-correlation. The derived signal processing procedure was verified in experiments on a polyamide specimen. Here, a ranging accuracy of <0.1 mm and a transmission ToF accuracy of 0.3μs were achieved. Thus, the proposed method enables fast and accurate non-invasive ToF measurements that do not require knowledge about transducer characteristics or ambient conditions.