The formation of close binaries has been an open question for decades. A large fraction of close binaries are in triple systems, suggesting that their formation may be associated with the Kozai-Lidov mechanism. However, this picture remains under debate because the configurations of many observed triples are unlikely to trigger the Kozai-Lidov mechanism. In this paper, we use the close binary samples, including eclipsing, spectroscopic, and astrometric binaries, from Gaia Data Release 3 to investigate the mysterious connection between inner binaries and their wide tertiaries. We show that the wide tertiary (at 10 3 -10 4 AU) fraction increases with decreasing orbital periods of the inner binaries. Compared to the field wide binary fraction, the wide tertiary fraction is 2.28 ± 0.10 times higher for eclipsing binaries (a median orbital period of 0.44 day) and 0.65 ± 0.03 times lower for astrometric binaries (a median orbital period of 537 days). The separation distribution of wide tertiaries is similar to wide binaries, with a tentative excess at ∼ 10 4 AU for tertiaries of eclipsing binaries. Using the v-r angle distributions, we show that the wide tertiaries are consistent with isotropic orientations with respect to the inner binaries. The inferred eccentricity distribution of wide tertiaries is close to thermal (f (e) ∝ e), similar to wide binaries at similar separations. The dynamical unfolding scenario is disfavored because it predicts highly eccentric wide tertiaries, which is inconsistent with our findings. For the Kozai-Lidov mechanism to be effective for wide tertiaries at > 10 3 AU, the initial separations of the inner binaries need to be > 3 AU. Future theoretical investigations are needed to explore the parameter space at these large initial separations and large tertiary separations.