The Internet of Things (IoT) provides a way to reach every hidden corner of the universe and provides valuable information for exploring the universe. With the advancement of hardware and computing technologies, IoT systems are developed based on systematically designed architectures. This paper discussed the routing protocol for low power lossy network (RPL) protocol's security vulnerabilities and proposed solutions. This review provides the main aspects of an IoT stack model with application, perception, and network layers. Then, the details and security concerns of the RPL protocol are given with the comparison of different frameworks. Different kinds of attacks, such as Wormhole, Sinkhole, and Denial of service attacks, are provided with the solution of defense‐based, threshold‐based, and trust‐supported based cryptography. Also, reviewed IoT security literature and focused on IoT security challenges and their solutions using emerging technologies like Machine Learning (ML), Blockchain, and Cloud Storage. A comparison and analysis of emerging technologies in the context of IoT security are also discussed. Due to the vulnerability of IoT devices, the scale of deployment, the attacks and suggested solutions are increased with scope and scale. In addition, related RPL‐based protocols are compared regarding energy efficiency, flexibility, scalability, robustness, and security. In the end, research challenges are identified, and future research direction for IoT security is suggested.