In the rapidly evolving landscape of the Internet of Things (IoT), cybersecurity remains a critical challenge due to the diverse and complex nature of network traffic and the increasing sophistication of cyber threats. This study investigates the application of the Artificial Bee Colony (ABC) algorithm for hyperparameter optimization (HPO) in machine learning classifiers, specifically focusing on Decision Trees, Support Vector Machines (SVM), and K-Nearest Neighbors (KNN) for IoT network traffic analysis and malware detection. Initially, the basic machine learning models demonstrated accuracies ranging from 69.68% to 99.07%, reflecting their limitations in fully adapting to the varied IoT environments. Through the employment of the ABC algorithm for HPO, significant improvements were achieved, with optimized classifiers reaching up to 100% accuracy, precision, recall, and F1-scores in both training and testing stages. These results highlight the profound impact of HPO in refining model decision boundaries, reducing overfitting, and enhancing generalization capabilities, thereby contributing to the development of more robust and adaptive security frameworks for IoT environments. This study further demonstrates the ABC algorithm’s generalizability across different IoT networks and threats, positioning it as a valuable tool for advancing cybersecurity in increasingly complex IoT ecosystems.