We focus on the problems of the accurate time delay estimation, the design of training pilots, and hybrid matrix optimization within the large‐scale antenna array Terahertz (THz) broadband communication system. In contrast to the existing researches based on narrow‐band arrays, we hereby shed light on the time delay estimation of broadband arrays. In THz broadband communication systems, the data symbol duration is relatively short when comparing with the dimension of the antenna array. In large‐scale antenna systems, signals received in each antenna are no longer different phase‐shifted copies of the same symbol, but completely different symbols in which occasion traditional narrow‐band structure is no longer suitable. Based on the above conclusion, firstly, we put forward a system model based on large‐scale antenna arrays and Time delay line (TDL) structure. Secondly, we deduce the Cramer‐Rao lower bound (CRLB) of the time delay estimation, and present a time delay estimation algorithm that could reach the CRLB. Thirdly, by minimizing the CRLB, we address the design of the training pilot and optimized TDL structure under the condition of constant envelope training pilot and modulus TDL structure. Finally, we disclose the numerical simulation results. According to the simulation results, the aforementioned method is workable in reaching the CRLB, the TDL structure can significantly surpass that of the traditional model, and the optimal pilot design method outperforms the pseudo‐random pilot structure.