Aiming at unknown or variant ransomware attack encrypted with SSL (Secure Sockets Layer)/ TLS (Transport Layer Security) protocol, a detection framework named TGAN-IDS (Transferred Generating Adversarial Network-Intrusion Detection System) based on dual generative adversarial networks is presented in this paper. In this framework, DCGAN (Deep Convolutional Generative Adversarial Network) is adopted to train a generator which has good performance to generate adversarial sample, and is transferred to the generator of TGAN. A pre-training model named PreD is built based on CNN (Convolutional Neural Network), which has good performance to do binary classification, and is transferred to the discriminator of TGAN. The generator and discriminator of TGAN play games in training process until the discriminator has a strong ability to detection unknown attack, and then it is output as an anomaly detector. In order to suppress the deterioration of normal sample detection ability during adversarial training of TGAN, a reconstruction loss function is introduced into the target function of TGAN. Experiments on a mixed dataset which is constructed by CICIDS2017 and other ransomware datasets show comparing with other deep learning network, such as AlexNet, ResNet and DenseNet etc., TGAN-IDS performs well in the indicators of detection accuracy, recall or F1-score etc. Also experiments on KDD99, SWaT and WADI datasets show that TGAN-IDS is suitable for other unencrypted unknown network attack detection.INDEX TERMS Ransomware, encrypted traffic, anomaly detection, GAN, transfer learning.

Detecting Android malware in its spread or download stage is a challenging work, which can realize early detection of malware before it reaches user side. In this paper, we propose a two-stage detection framework based on feature enhancement and cascade deep forest. This method can detect the traffic generated in the encrypted transmission process of Android malware. The first stage realizes the binary classification of benign and malicious software. The second stage realizes the multi-classification of different categories of malware. To enhance data representation, convolutional neural networks is used to extract benign and malicious features in the first stage, and the principal component analysis method is used to extract the malicious features in the second stage. Theses extracted features are spliced with the payload part of the traffic to form fusion features for classification task. In order to adapt to different scale of samples, especially for the small-scale sample, cascaded deep forest method is proposed to construct the classification model. In this model, many layers that consist of base classifiers are cascaded and the number of layers can be automatically adjusted according to the scale of the samples. With different combinations of base classifiers in each layer, the optima detection accuracy is archived in the two stages. The experimental results on several datasets prove that the proposed method is effective for encrypted transmission detection of Android malware. It is also suitable for the detection of unknown attacks.