The Internet of Things (IoT) is a collection of devices such as sensors for collecting data, actuators that perform mechanical actions on the sensor's collected data, and gateways used as an interface for effective communication with the external world. The IoT has been successfully applied to various fields, from small households to large industries. The IoT environment consists of heterogeneous networks and billions of devices increasing daily, making the system more complex and this need for privacy and security of IoT devices become a major concern. The critical components of IoT are device identification, a large number of sensors, hardware operating systems, and IoT semantics and services. The layers of a core IoT application are presented in this paper with the protocols used in each layer. The security challenges at various IoT layers are unveiled in this review paper along with the existing mitigation strategies such as machine learning, deep learning, lightweight encryption techniques, and Intrusion Detection Systems (IDS) to overcome these security challenges and future scope. It has been concluded after doing an intensive review that Spoofing and Distributed Denial of Service (DDoS) attacks are two of the most common attacks in IoT applications. While spoofing tricks systems by impersonating devices, DDoS attacks flood IoT systems with traffic. IoT security is also compromised by other attacks, such as botnet attacks, man-in-middle attacks etc. which call for strong defenses including IDS framework, deep neural networks, and multifactor authentication system.