Gas-filled hollow core fibers allow the generation of single-cycle pulses at megahertz repetition rates. When coupled with difference frequency generation, they can be an ideal driver for generating carrier-envelope phase stable, octave-spanning pulses in the short-wavelength infrared. In this work, we investigate the dependence of the polarization state in gas-filled hollow-core fibers (HCF) on the subsequent difference frequency generation stage. We show that by adjusting the input polarization state of light in geometrically symmetric systems, such as hollow-core fibers, one can achieve precise control over the polarization state of the output pulses. This manipulation preserves the temporal characteristics of the generated ultrashort pulses, especially when operating at a near single-cycle regime. We leverage this property to boost the downconversion efficiency of the near single-cycle pulses in a type I difference frequency generation stage. Our technique overcomes the bandwidth and dispersion constraints of the previous methods that rely on broadband waveplates or adjustment of crystal axes relative to the laboratory frame. This advancement is crucial for experiments demanding pure polarization states in the eigenmodes of the laboratory frame.