We study polarization effects in the nonlinear interference of photons generated via frequency non-degenerate spontaneous parametric down conversion. Signal and idler photons generated in the visible and infrared (IR) range, are split in different arms of a nonlinear Michelson interferometer. The interference pattern for signal photons is detected, and it is shown to be dependent on the polarization rotation of idler photons, introduced by a birefringent sample. Based on this concept, we realize two new methods for measurement of sample retardation in the IR range by using well-developed and inexpensive components for visible light. The accuracy of the methods meets current industry standards. The developed IR polarimetry technique is relevant to material research, optical inspection, and quality control.