We present a polarization diverse receiver architecture for continuous variables quantum key distribution. Our receiver architecture forgoes the need for any polarization matching calibrations or feedback loops and is usable in any locally generated local oscillator key distribution system. We demonstrate experimentally that our system is capable of reliably achieving theoretical security against coherent attacks, even under very adverse random polarization drift scenarios, where the single polarization channels are unable to achieve security, and is capable of functioning for long periods of time.