Technologically speaking, humanity lives in an age of evolution, prosperity, and great development, as a new generation of the Internet has emerged; it is the Internet of Things (IoT) which controls all aspects of lives, from the different devices of the home to the large industries. Despite the tremendous benefits offered by IoT, still there are some challenges regarding privacy and information security. The traditional techniques used in Malware Anomaly Detection Systems (MADS) could not give us as robust protection as we need in IoT environments. Therefore, it needed to be replaced with Deep Learning (DL) techniques to improve the MADS and provide the intelligence solutions to protect against malware, attacks, and intrusions, in order to preserve the privacy of users and increase their confidence in and dependence on IoT systems. This research presents a comprehensive study on security solutions in IoT applications, Intrusion Detection Systems (IDS), Malware Detection Systems (MDS), and the role of artificial intelligent (AI) in improving security in IoT.