We discuss the role of gluon poles and the gauge invariance for the hadron tensors of Drell-Yan and direct photon production processes with the transversely polarized hadron. These hadron tensors are needed to construct the corresponding single spin asymmetries. For the Drell-Yan process, we perform our analysis within both the Feynman and axial-type (contour) gauges for gluons. In both the Feynman and contour gauges, we demonstrate that the gauge invariance leads to the need of the new (non-standard) diagrams. Moreover, in the Feynman gauge, we argue the absence of gluon poles in the correlators ψ γ ⊥ A + ψ related traditionally to dT (x, x)/dx. As a result, these terms disappear from the final QED gauge invariant Drell-Yan hadron tensor. For the direct photon production, by using the contour gauge for gluon fields, we find that there are new twist-3 terms present in the hadron tensor of the considering process in addition to the standard twist-3 terms.