Privacy protection is an essential part of information security. The use of shared resources demands more privacy and security protection, especially in cloud computing environments. Side-channel attacks based on CPU cache utilize shared CPU caches within the same physical device to compromise the system’s privacy (encryption keys, program status, etc.). Information is leaked through channels that are not intended to transmit information, jeopardizing system security. These attacks have the characteristics of both high concealment and high risk. Despite the improvement in architecture, which makes it more difficult to launch system intrusion and privacy leakage through traditional methods, side-channel attacks ignore those defenses because of the shared hardware. Difficult to be detected, they are much more dangerous in modern computer systems. Although some researchers focus on the survey of side-channel attacks, their study is limited to cryptographic modules such as Elliptic Curve Cryptosystems. All the discussions are based on real-world applications (e.g., Curve25519), and there is no systematic analysis for the related attack and security model. Firstly, this paper compares different types of cache-based side-channel attacks. Based on the comparison, a security model is proposed. The model describes the attacks from four key aspects, namely, vulnerability, cache type, pattern, and range. Through reviewing the corresponding defense methods, it reveals from which perspective defense strategies are effective for side-channel attacks. Finally, the challenges and research trends of CPU cache-based side-channel attacks in both attacking and defending are explored. The systematic analysis of CPU cache-based side-channel attacks highlights the fact that these attacks are more dangerous than expected. We believe our survey would draw developers’ attention to side-channel attacks and help to reduce the attack surface in the future.