The continuous improvement in the connectivity, automation and autonomy levels of Intelligent Connected Vehicles (ICVs) significantly increases the probability of potential security threats. Over-the-Air (OTA) is a promising technique for upgrading features of ICVs and enhancing their reliability and security against environmental disturbances as well as malicious attacks. To better understand the potential security risks and possible countermeasures, we survey research works in ICV security during OTA from cloud upgrade, terminal upgrade, and object upgrade. We also summarize existing methods in OTA upgrading techniques and systematically investigate the overall framework of OTA upgrading methods from the perspectives of Software-Over-the-Air (SOTA) and Firmware-Over-the-Air (FOTA).We further discuss possible mitigation strategies and open issues yet to be resolved in this research direction. This survey shows that OTA provides a powerful technique for upgrading the ICV features and improving ICV security.