To acquire the enhanced underwater ship-radiated noise signal in the presence of array shape distortion in a passive sonar system, the phase difference of the line-spectrum component in ship-radiated noise is often exploited to estimate the time-delay difference for the beamforming-based signal enhancement. However, the time-delay difference estimation performance drastically degrades with decreases of the signal-to-noise ratio (SNR) of the line-spectrum component. Meanwhile, although the time-delay difference estimation performance of the high-frequency line-spectrum components is generally superior to that of the low-frequency one, the phase difference measurements of the high-frequency line-spectrum component often easily encounter the issue of modulus 2π ambiguity. To address the above issues, a novel time-frequency joint time-delay difference estimation method is proposed in this paper. The proposed method establishes a data-driven hidden Markov model with robustness to phase difference ambiguity by fully exploiting the underlying property of slowly changing the time-delay difference over time. Thus, the phase difference measurements available for time-delay difference estimation are extended from that of low-frequency line-spectrum components in a single frame to that of all detected line-spectrum components in multiple frames. By jointly taking advantage of the phase difference measurements in both time and frequency dimensions, the proposed method can acquire enhanced time-delay difference estimates even in a low SNR case. Both simulation and at-sea experimental results have demonstrated the effectiveness of the proposed method.