The contemporary era places paramount importance on network security and cloud environments, driven by increased data transmission demands, the flexibility of cloud services, and the prevalence of global resources. Addressing the escalating threat of computer malware, the development of efficient intrusion detection systems (IDS) is imperative. This research focuses on the challenges posed by imbalanced datasets and the necessity for unsupervised learning to enhance network security. The proposed hybrid deep learning method utilizes raw data from the CSE-CIC-IDS-2018 dataset, integrating imbalanced and unsupervised learning techniques. After preprocessing and normalization, feature extraction through principal component analysis (PCA) reduces dimensionality from seventy-eight fields to ten essential features. Clustering, employing the density-based spatial clustering of applications with noise (DBSCAN) algorithm optimized with particle swarm optimization (PSO), is applied to the extracted features, distinguishing between attack and non-attack packets. Addressing dataset imbalances, imbalanced learning techniques are employed, and unsupervised learning is exemplified through the AutoEncoder (AE) algorithm. The attack cluster’s data is input into AE, a deep learning-based approach, yielding outputs for attack classification. The proposed technique (PCA+DBSCANPSO+AE) achieves an impressive 99.19% accuracy in intrusion detection, surpassing contemporary methodologies and five existing techniques. This research not only enhances accuracy but also addresses imbalanced learning challenges, utilizing the power of unsupervised learning for robust network security.