We study the influence of the polarization states of ionizing femtosecond two-color pulses on the emitted terahertz radiation in gases. A local-current model and plane-wave evaluations justify the previously-reported impact on the THz energy yield and a (almost) linearly-polarized THz field when using circularly-polarized laser harmonics. For such pump pulses, the THz yield is independent of the relative phase between the two colors. When the pump pulses have same helicity, the increase in the THz yield is associated with longer ionization sequences and higher electron transverse momenta acquired in the driving field. Reversely, for two color pulses with opposite helicity, the dramatic loss of THz power comes from destructive interferences driven by the highly symmetric response of the photocurrents lined up on the third harmonic of the fundamental pulse. While our experiments confirm an increased THz yield for circularly-polarized pumps of same helicity, surprisingly, the emitted THz radiation is not linearly-polarized. This effect is explained by means of comprehensive 3D numerical simulations highlighting the role of the spatial alignment and non-collinear propagation of the two colors.