Intrusion Detection Systems (IDS) play a major part in protecting security threats and networks from attacks. Due to the rapid development of the internet of things (IoT), more cyber‐attacks are attacking these devices. Various security challenges still occur on IoT devices since most of them have limited security mechanisms. Hence, this paper introduces a combination of linear and non‐linear space transformation models for IDS. Independent component analysis (ICA) is employed for linear transformation to obtain an orthogonal space, and a dual‐phase distance metric learning method (D‐DML) is utilized to obtain an optimal distance metric. The Gaussian radial basis function (GRBF) model is employed for non‐linear transformation. Then these features of linear and non‐linear models are integrated and classified by capsule auto encoder with a hybrid kernel function (HKCAE) which classifies normal and malicious attacks. Guidance of the capuchin search algorithm (CSA) is employed to optimize the HKCAE parameters during prediction attack prediction. The performance of the implemented approach is compared with the other approaches with some measures like precision, accuracy, sensitivity, F‐score, and specificity benchamrked through UNSW‐15 and BoT‐IoT datasets. The accuracy of the developed scheme is 0.9973 and 0.999 on UNSW‐15 and BoT‐IoT datasets, respectively.