Unsupervised Anomaly Detection for Intricate KPIs via Adversarial Training of VAE

Chen, Wenxiao; Xu, Haowen; Li, Zeyan; Pei, Dan; Chen, Jie; Qiao, Huiting; Feng, Yang; Wang, Zhaogang

doi:10.1109/infocom.2019.8737430

Cited by 80 publications

(18 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Donut puts forward innovations, such as M-ELBO, MCMC iteration, and missing value zero fillings on the basis of VAE, which has excellent performance on periodic KPIs. Subsequently, Buzz [6] solved the problem that was difficult for donut as it handles more complex data distribution. It measures the distance of data distributions and generates distributions through the Wasserstein distance.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Novel Hybrid Method for KPI Anomaly Detection Based on VAE and SVDD

et al. 2021

View full text Add to dashboard Cite

Key performance indicator (KPI) anomaly detection is the underlying core technology in Artificial Intelligence for IT operations (AIOps). It has an important impact on subsequent anomaly location and root cause analysis. Variational auto-encoder (VAE) is a symmetry network structure composed of encoder and decoder, which has attracted extensive attention because of its ability to capture complex KPI data features and better detection results. However, VAE is not well applied to the modeling of KPI time series data and it is often necessary to set the threshold to obtain more accurate results. In response to these problems, this paper proposes a novel hybrid method for KPI anomaly detection based on VAE and support vector data description (SVDD). This method consists of two modules: a VAE reconstructor and SVDD anomaly detector. In the VAE reconstruction module, firstly, bi-directional long short-term memory (BiLSTM) is used to replace the traditional feedforward neural network in VAE to capture the time correlation of sequences; then, batch normalization is used at the output of the encoder to prevent the disappearance of KL (Kullback–Leibler) divergence, which prevents ignoring latent variables to reconstruct data directly. Finally, exponentially weighted moving average (EWMA) is used to smooth the reconstruction error, which reduces false positives and false negatives during the detection process. In the SVDD anomaly detection module, smoothed reconstruction errors are introduced into the SVDD for training to determine the threshold of adaptively anomaly detection. Experimental results on the public dataset show that this method has a better detection effect than baseline methods.

show abstract

Section: Related Workmentioning

confidence: 99%

“…(1) VAE is not well suited for time series modeling. Previous VAE-based KPI anomaly detection methods [5,6] regard time series as sliding windows, ignoring the time relationship between sliding windows in the encoding process. In order to solve this problem, researchers combine LSTM [7] and VAE.…”

Section: Introductionmentioning

confidence: 99%

A Novel Hybrid Method for KPI Anomaly Detection Based on VAE and SVDD

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Hence, supervised machine learning strategies like [38][39][40] that depend on prior patterns of normal and abnormal behaviours are not suitable for real-time anomaly detection systems. Many recent works like [33,41], propose unsupervised machine learning algorithms for detecting anomalies. However, these unsupervised machine learning approaches train the models offline and hence, they cannot effectively adapt to the rapidly changing network behaviours over time and learn new threats and vulnerabilities.…”

Section: Real-time Security With Unsupervised Deep Learning Modelsmentioning

confidence: 99%

“…However, kitsune adds a separate feature extraction step which requires a basic understanding of the underlying network protocols. In [41], authors use adversarial training of VAE to detect anomalies in key performance monitors of the network however, it unsuitable for online systems as the adversarial training is unstable and difficult to converge. Khatuya et al [121] propose ADELE, with an aim to select features from system logs in order to create groundwork for a proactive, online failure prediction system.…”

Section: Unsupervised Learning Approachesmentioning

confidence: 99%

“…This enables the system administrators to intervene in an on-going attack or fix a system performance issue. In this regard, offline learning strategies like [38,41,44,117], that require the knowledge of prior pattern of normal and abnormal events are not suitable for the new detection systems and also for existing systems trained on outdated events.…”

Section: Unsupervised Learning Approachesmentioning

confidence: 99%

See 1 more Smart Citation

PHOENIX: A Premise to Reinforce Heterogeneous and Evolving Internet Architectures with Exemplary Applications

Adhatarao¹

View full text Add to dashboard Cite

The Internet was designed primarily for sharing expensive resources through establishing point-to-point connection using dedicated IP addresses. However, it has since evolved both in complexity and its number of users. Subsequently, various heterogeneous networking architectures and protocols have emerged to fulfill the growing needs of users. Information-Centric Networking (ICN) is a promising new networking paradigm which treats content as the first-class entity. In ICN, nodes exchange information based on the Names of the content instead of the IP addresses of the end points that request or provide the information. Internet of Things (IoT) has gained popularity in recent years and billions of devices are expected to connect in the near future. With advancements in technology and availability of Big data, Artificial Intelligence is producing promising new applications and services. Additionally, many new architectures and services like data centers, Cloud computing, Edge/Fog computing, etc., have also emerged to enhance the Internet by providing cost-effective, flexible and scalable computing platforms for the growing number of applications. However, nowadays, users are more interested in retrieving content(s) of their interest irrespective of its location. While, devices in the Sensor Networks need a larger address space and low communication overhead due to constrained resources. In addition, machine learning applications demand high computing power and often applications have to find a trade-off between resource consumption and desired accuracy. Moreover, these heterogeneous networks have different requirements and hence, they operate with dissimilar protocols. Additionally, there are many important open issues within these heterogeneous networks like naming and mobility in ICN, pub/sub and protocol translation in IoT, optimizing resource-accuracy trade off with machine learning, etc., that need to be resolved. This dissertation analyzes the heterogeneous and evolving Internet architectures research space and identifies six main open issues and reinforces the Internet with efficient and optimal solutions. In particular, the dissertation analyzes the two most important naming schemas in ICN: hierarchical and flat names through examining the various interlinked metrics and proposes the optimal choices for efficient naming schemas at the application and network layers in ICN. This is followed by a proposal for a gateway and additional functions to seamlessly integrate the heterogeneous Sensor Networks to the Internet to realize IoT. Additionally, we study the requirements of resource-constrained IoT devices and provide an efficient lightweight pub/sub system to optimize the resource utilization in IoT networks. We further provide a comprehensive and robust solution to support the network mobility in ICN iv through a distributed Mobility Agent service architecture followed by two distinct frameworks to address the open issues in deep learning 1) EVA: which provides a distributed architecture to optimize the reso...

show abstract

Dynamic threshold strategy optimization for security protection in Internet of Things: An adversarial deep learning‐based game‐theoretical approach

Chen

Wang

Guo

et al. 2022

Concurrency and Computation

View full text Add to dashboard Cite

As mobile communications, the Internet, databases, distributed computing, and other technologies continue to develop, the Internet of Things (IoT) has emerged as prevalent technique. However, attacks on security and sensitive data in IoT occur frequently, and these attacks often evade intrusion detection systems strategically by mutating their traffic. To prevent security threats and sensitive data leakage, we propose a game approach based on adversarial deep learning to optimize a dynamic security threshold strategy. We introduce a mobile edge computing framework and utilize a game model to describe the adversarial interaction between the two participants. To solve the complexity of the game problem to gain dynamically randomized adversarial attacks, we present a column generation (CG) framework, which uses a feedforward neural network to quantify data flowing through IoT devices. Considering the limited resources of IoT devices, we calculate an optimal response to cyberattacks via a particle swarm optimization algorithm, aiming to reduce the false alarm rate. The adversarial dynamic threshold (ADT)‐based column generation (CG‐ADT) algorithm generates the set of detection threshold and the probability. Finally, we present the results of experiments conducted to demonstrate the effectiveness and robustness of the proposed dynamic threshold scheme for sensitive data security protection in IoT and its suitability for implementation in production systems.

show abstract

Unsupervised Anomaly Detection for Intricate KPIs via Adversarial Training of VAE

Cited by 80 publications

References 14 publications

A Novel Hybrid Method for KPI Anomaly Detection Based on VAE and SVDD

A Novel Hybrid Method for KPI Anomaly Detection Based on VAE and SVDD

PHOENIX: A Premise to Reinforce Heterogeneous and Evolving Internet Architectures with Exemplary Applications

Dynamic threshold strategy optimization for security protection in Internet of Things: An adversarial deep learning‐based game‐theoretical approach

Contact Info

Product

Resources

About