Time Delay Integration (TDI) is an effective approach for high sensitive infrared detectors. According to the principle of the TDI, the central distance of pixel along the time delay integral direction is closely linked with the specific application requirements. So the optimization design, such as the area of pixels and their distance, plays an important role to improve the performance of TDI detectors. The crosstalk between pixels is a crucial factor that results in the decline of detector Modulation Transfer Function (MTF), and then affects the imaging quality. In this paper the optimization design rule for the arrangement of pixel has been investigated. The results show that the main method to appreciate it is reducing the crosstalk between pixels and enhancing detectivity. Chips of which pixel areas and edge intervals are different but with same distance were designed for the experiments. The optical and electrical measurements were carried out for these chips and the optimized structure was obtained. In addition, relationships between the crosstalk and parameters of material, pixel structure were analyzed based on the experiment data. According to the comprehensive analysis of the measurement data, we obtained the optimum design for specific HgCdTe TDI infrared detector. Meanwhile it is also a well reference for other HgCdTe TDI detector structure design.