Accurate Recovery of Missing Network Measurement Data With Localized Tensor Completion

Xie, Kun; Wang, Xiangge; Wang, Xin; Chen, Yuxiang; Xie, Gaogang; Ouyang, Yudian; Wen, Junhai; Cao, Jiannong; Zhang, Dafang

doi:10.1109/tnet.2019.2940147

Cited by 33 publications

(11 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…IV. DOWNSTREAM APPLICATIONS Although there are many proposed methods for imputing or recovering missing values in the network data [1], [4], most of them concentrated on improving the imputation error without considering the performance of downstream applications. In this section, we present how Traffic Engineering leverages network traffic imputation.…”

Section: Spatial Feature Learning With Gcnmentioning

confidence: 99%

“…Matrix and tensor factorization were also applied to solve the data imputation problem. Many tensor completion algorithms have been proposed based on Alternating Least Square (ALS) such as Localized Tensor Decomposition (LTC) [4], gradient-based method such as Generalized Canonical Polyadic Tensor Decomposition (GCP) [5]. Among those, LTC [4] would be seen as the most recent work on estimating missing values, tailored for network traffic data.…”

Section: Introductionmentioning

confidence: 99%

“…Many tensor completion algorithms have been proposed based on Alternating Least Square (ALS) such as Localized Tensor Decomposition (LTC) [4], gradient-based method such as Generalized Canonical Polyadic Tensor Decomposition (GCP) [5]. Among those, LTC [4] would be seen as the most recent work on estimating missing values, tailored for network traffic data. In LTC, the traffic matrix was represented as a 3-way tensor, including days, hours, and origin-destination dimension, to exploit the inherent relationship among higherdimensional data.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

GCRINT: Network Traffic Imputation Using Graph Convolutional Recurrent Neural Network

Nguyen

et al. 2021

ICC 2021 - IEEE International Conference on Communications

View full text Add to dashboard Cite

Missing values appear in most multivariate time series, especially in the monitored network traffic data due to high measurement cost and unavoidable loss. In the networking fields, missing data prevents advanced analysis and downgrades downstream applications such as traffic engineering and anomaly detection. Despite the great potential, existing imputation approaches based on tensor decomposition and deep learning techniques have shown limitations in addressing missing values of traffic data due to its dynamic behavior. In this paper, we propose Graph Convolutional Recurrent Neural Network for Imputing Network Traffic (GCRINT), a combination between Recurrent Neural Network (RNN) and Graph Convolutional Neural Network, for filling the missing values of network traffic data. We use a bidirectional Long Short-Term Memory network and Graph Neural Network to efficiently learn the spatial-temporal correlations in partially observed data. We conducted extensive experiments to evaluate our model by using two different datasets and various missing scenarios. The experiment results show that GCRINT achieves significantly low imputation errors and reduces the error by 35% compared to the state-of-the-art methods. GCRINT also helps to obtain a stable performance in the traffic engineering problem.

show abstract

Section: Spatial Feature Learning With Gcnmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

GCRINT: Network Traffic Imputation Using Graph Convolutional Recurrent Neural Network

Nguyen

et al. 2021

ICC 2021 - IEEE International Conference on Communications

View full text Add to dashboard Cite

show abstract

“…Such growth has, in turn, resulted in significant challenges for network resource orchestration, capacity planning, service provisioning, and traffic engineering. One of the most important input factors in these tasks is traffic data [4,12,20,26,29,39]. It comprises the volumes of traffics in bytes, packets, or flows during a specified period of time between Origin and Destination (OD) pairs which can be edge routers in the WAN network or the ToR switches in the data center settings in the network [31].…”

Section: Introductionmentioning

confidence: 99%

A Convergent Semi-Proximal Alternating Direction Method of Multipliers for Recovering Internet Traffics from Link Measurements

Ming,

Zhang,

et al. 2021

Preprint

View full text Add to dashboard Cite

It is challenging to recover the large-scale internet traffic data purely from the link measurements. With the rapid growth of the problem scale, it will be extremely difficult to sustain the recovery accuracy and the computational cost. In this work, we propose a new Sparsity Low-Rank Recovery (SLRR) model and its Schur Complement Based semi-proximal Alternating Direction Method of Multipliers (SCB-spADMM) solver. Our approach distinguishes itself mainly for the following two aspects. First, we fully exploit the spatial low-rank property and the sparsity of traffic data, which are barely considered in the literature. Our model can be divided into a series of subproblems, which only relate to the traffics in a certain individual time interval. Thus, the model scale is significantly reduced. Second, we establish a globally convergent ADMM-type algorithm inspired by [Li et al., Math. Program., 155(2016)] to solve the SLRR model. In each iteration, all the intermediate variables' optimums can be calculated analytically, which makes the algorithm fast and accurate. Besides, due to the separability of the SLRR model, it is possible to design a parallel algorithm to further reduce computational time. According to the numerical results on the classic datasets Abilene and GEANT, our method achieves the best accuracy with a low computational cost. Moreover, in our newly released large-scale Huawei Origin-Destination (HOD) network traffics, our method perfectly reaches the seconds-level feedback, which meets the essential requirement for practical scenarios.

show abstract

“…And in recommendation systems [5], [6], LRTC is utilized to select relevant information. In communications [7], [8], LRTC can infer the remaining network monitoring data by leveraging their spatial-temporal correlations when a subset of paths or time intervals of the network can be measured. And in biomedical data analysis [9], [10], LRTC is also used to improve spatial-temporal resolution of magnetic resonance imaging data by reconstructing the whole data from very few measurements.…”

mentioning

confidence: 99%

Multilayer Sparsity-Based Tensor Decomposition for Low-Rank Tensor Completion

Xue

Zhao

Huang

et al. 2022

IEEE Trans. Neural Netw. Learning Syst.

136

View full text Add to dashboard Cite

Existing methods for tensor completion (TC) have limited ability for characterizing low-rank structures. To depict the complex hierarchical knowledge with implicit sparsity attributes hidden in a tensor, we propose a new multi-layer sparsity-based tensor decomposition (MLSTD) for low-rank tensor completion (LRTC). The method encodes structured sparsity of a tensor by multiple layer representation. Specifically, we use the CANDECOMP/PARAFAC (CP) model to decompose a tensor into an ensemble of sum of rank-1 tensors, and the number of rank-1 component is easily interpreted as a first layer sparsity measure. Presumably, the factor matrices are smooth since local piece-wise property exists in within-mode correlation. In subspace, the local smoothness can be regarded as the second layer sparsity. To describe the refined structures of factor/subspace sparsity, we introduce a new sparsity insight of subspace smoothness: a self-adaptive low-rank matrix factorization scheme, called the third layer sparsity. By progressive description of sparsity structure, we formulate a MLSTD model and embed it into the LRTC problem. Then an effective ADMM algorithm is designed for the MLSTD minimization problem. Various experiments in RGB images, hyperspectral images and videos substantiate the proposed LRTC method is superior to state-of-the-art methods.

show abstract

Accurate Recovery of Missing Network Measurement Data With Localized Tensor Completion

Cited by 33 publications

References 48 publications

GCRINT: Network Traffic Imputation Using Graph Convolutional Recurrent Neural Network

GCRINT: Network Traffic Imputation Using Graph Convolutional Recurrent Neural Network

A Convergent Semi-Proximal Alternating Direction Method of Multipliers for Recovering Internet Traffics from Link Measurements

Multilayer Sparsity-Based Tensor Decomposition for Low-Rank Tensor Completion

Contact Info

Product

Resources

About