HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Network covert channels break a computer's security policy to establish a stealthy communication. They are a threat being increasingly used by malicious software. Most previous studies on detecting network covert channels using Machine Learning (ML) were tested with a dataset that was created using one single covert channel tool and also are ineffective at classifying covert channels into patterns. In this paper, selected ML methods are applied to detect popular network covert channels. The capacity of detecting and classifying covert channels with high precision is demonstrated. A dataset was created from nine standard covert channel tools and the covert channels are then accordingly classified into patterns and labelled. Half of the generated dataset is used to train three different ML algorithms. The remaining half is used to verify the algorithms' performance. The tested ML algorithms are Support Vector Machines (SVM), k-Nearest Neighbors (k-NN) and Deep Neural Networks (DNN). The k-NN model demonstrated the highest precision rate at 98% detection of a given covert channel and with a low false positive rate of 1%.

The detection and elimination of covert channels are performed by a network node, known as a warden. Especially if faced with adaptive covert communication parties, a regular warden equipped with a static set of normalization rules is ineffective compared to a dynamic warden. However, dynamic wardens rely on periodically changing rule sets and have their own limitations, since they do not consider traffic specifics. We propose a novel adaptive warden strategy, capable of selecting active normalization rules by taking into account the characteristics of the observed network traffic. Our goal is to disturb the covert channel and provoke the covert peers to expose themselves more by increasing the number of packets required to perform a successful covert data transfer. Our evaluation revealed that the adaptive warden has better efficiency and effectiveness when compared to the dynamic warden because of its adaptive selection of normalization rules.