We present a new method for time delay estimation using band limited frequency domain data representing the port responses of interconnect structures. The approach is based on the spectrally accurate method for causality characterization that employs SVD-based causal Fourier continuations, which was recently developed by the authors. The time delay extraction is constructed by incorporating a linearly varying phase factor to the system of equations that determines the Fourier coefficients. The method is capable of determining the time delay using data affected by noise or approximation errors that come from measurements or numerical simulations. It can also be employed when only a limited number of frequency responses is available. The technique can be extended to multi-port and mixed-mode networks. Several analytical and simulated examples are used to demonstrate the accuracy and strength of the proposed technique.