The Internet of Things (IoT) represents a paradigm shift in which the Internet is connected to real objects in a range of areas, including home automation, industrial processes, human health, and environmental monitoring. The global market for IoT devices is booming, and it is estimated that there will be 50 billion connected devices by the end of 2025. This explosion of IoT devices, which can be expanded more easily than desktop PCs, has led to an increase in cyber-attacks involving IoT devices. To address this issue, it is necessary to create novel approaches for identifying attacks launched by hacked IoT devices. Due to the possibility that these attacks would succeed, Intrusion Detection Systems (IDS) are required. IDS' feature selection stage is widely regarded as the most essential stage. This stage is extremely time-consuming and labor-intensive. However, numerous machine learning (ML) algorithms have been proposed to enhance this stage to boost an IDS's performance. These approaches, however, did not produce desirable results in terms of accuracy and detection rate (DR). In this paper, we propose a novel hybrid Autoencoder and Modified Particle Swarm Optimization (HAEMPSO) for feature selection and deep neural network (DNN) for classification. The PSO with modification of inertia weight was utilized to optimize the parameters of DNN. The experimental analysis was performed on two realistic UNSW-NB15 and BoT-IoT datasets that are suitable for IoT environment. The findings obtained by analyzing the proposed HAEMPSO against the Generic attack in the UNSW-NB15 dataset gave an accuracy of 98.8%, and a DR of 99.9%. While the benign class revealed an accuracy of 99.9% and DR of 99.7%. In the BoT-IoT dataset, the DDoS HTTP attack revealed an accuracy of 99.22% and DR of 97.79%. While the benign class gave an accuracy of 97.54% and DR of 97.92%. In comparison with the state-of-the-art machine learning schemes, our proposed HAEMPSO-DNN achieved a competitive feat in terms of DR and accuracy.