Decentralizing network inference problems with Multiple-Description Fusion Estimation (MDFE)

Abstract

Network inference (or tomography) problems, such as traffic matrix estimation or completion and link loss inference, have been studied rigorously in different networking applications. These problems are often posed as under-determined linear inverse (UDLI) problems and solved in a centralized manner, where all the measurements are collected at a central node, which then applies a variety of inference techniques to estimate the attributes of interest. This paper proposes a novel framework for decentralizing these large-scale under-determined network inference problems by intelligently partitioning it into smaller sub-problems and solving them independently and in parallel. The resulting estimates, referred to as multiple descriptions, can then be fused together to compute the global estimate. We apply this Multiple Description and Fusion Estimation (MDFE) framework to three classical problems: traffic matrix estimation, traffic matrix completion, and loss inference. Using real topologies and traces, we demonstrate how MDFE can speed up computation while maintaining (even improving) the estimation accuracy and how it enhances robustness against noise and failures. We also show that our MDFE framework is compatible with a variety of existing inference techniques used to solve the UDLI problems.

Decentralizing Network Inference Problems With Multiple-Description Fusion Estimation (MDFE)

“…In this paper and in [10], we demonstrate how MDFE can be applied to network inference problems such as TM Estimation (TME), TM Completion (TMC) and Loss Inference (LI), and we show, MDFE is compatible with different previously proposed inference techniques, including least square error estimation, expectation maximization, and regularized matrix factorization methods [8], [11], [4], [7].…”

“…Note that the CN of each individual row of H is one; however, this is not an interesting case because: 1) in practice, the number of processors/sub-spaces (L) are limited (in parallel case), and 2) large L's reduces processing gain ∆ s (in sequential case). Also, as we have shown in [10], the computational complexity of this algorithm is low. Thus, for a large-scale NI problem which is inefficient or impossible to be solved in a centralized manner, the MDFE approach with partitioning Alg.1 proposes an efficient framework, with manageable computational complexity and without compromising the accuracy of the solution.…”

“…Among these, Alg.1 and Alg.2 are more suitable in the centralized implementation of MDFE where all network measurements are available at the CFC, and Alg.3 is suitable for the distributed implementation of MDFE where measurements are collected by distributed nodes spread among the network, which compute local descriptions and transmit them to the CFC for the final reconstruction through fusion process. For further details of portioning algorithms, please refer to [10].…”

“…then total error variance is reduced by partitioning and 3) the lower-bound for the MSE of LNE is reduced by partitioning (For proof, please refer to supplementary Appendix A or [10]). …”

“…Finally, Section VI summarizes the main results of the paper. Due to space limitation, we refer to [10] for more detailed discussions and additional results.…”