The multi-layer information bottleneck problem

Yang, Qianqian; Piantanida, Pablo; Gündüz, Deniz

doi:10.1109/itw.2017.8278006

Cited by 10 publications

(6 citation statements)

References 13 publications

(15 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The bound I(f (X n ); g(Y n )) ≤ I(X; Y) was established, which is tight if and only if f and g are dictator functions. There are many other variations of multi-user IB in the literature [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25].…”

Section: Introductionmentioning

confidence: 99%

The Double-Sided Information-Bottleneck Function

Dikshtein

Ordentlich

Shitz

2021

2021 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

We consider a two-terminal variant (double-sided) of the information bottleneck problem, which is related to biclustering. In our setup, X and Y are dependent random variables and the problem is to find two independent channels P U|X and P V|Y (setting the Markovian structure U → X → Y → V) that maximize I(U; V) subject to constraints on the relevant mutual information expressions: I(U; X) and I(V; Y). For jointly Gaussian X and Y, we show that Gaussian channels are optimal in the low-SNR regime, but not for general SNR. Similarly, it is shown that for a doubly symmetric binary source, binary symmetric channels are optimal when the correlation is low, and are suboptimal for high correlation. We conjecture that Z and S channels are optimal when the correlation is 1 (i.e., X = Y), and provide supporting numerical evidence.

show abstract

Section: Introductionmentioning

confidence: 99%

The Double-Sided Information-Bottleneck Function

Dikshtein

Ordentlich

Shitz

2021

2021 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

show abstract

“…Seq2Seq models are also used extensively for the dynamic, spatial-temporal characteristic of multivariate time series data (Zhu and Laptev, 2017 ; Du et al, 2018 ; Yang et al, 2019 ; Salinas et al, 2020 ). Another application of the sequence to sequence model is in video and image captioning (Venugopalan et al, 2015 ; Yang et al, 2017 ).…”

Section: Time Series Prediction Methods For Mitigating Time Delay mentioning

confidence: 99%

A Brief Survey of Telerobotic Time Delay Mitigation

2020

View full text Add to dashboard Cite

There is a substantial number of telerobotics and teleoperation applications ranging from space operations, ground/aerial robotics, drive-by-wire systems to medical interventions. Major obstacles for such applications include latency, channel corruptions, and bandwidth which limit teleoperation efficacy. This survey reviews the time delay problem in teleoperation systems. We briefly review different solutions from early approaches which consist of control-theory-based models and user interface designs and focus on newer approaches developed since 2014. Future solutions to the time delay problem will likely be hybrid solutions which include modeling of user intent, prediction of robot movements, and time delay prediction all potentially using time series prediction methods. Hence, we examine methods that are primarily based on time series prediction. Recent prediction approaches take advantage of advances in nonlinear statistical models as well as machine learning and neural network techniques. We review Recurrent Neural Networks, Long Short-Term Memory, Sequence to Sequence, and Generative Adversarial Network models and examine each of these approaches for addressing time delay. As time delay is still an unsolved problem, we suggest some possible future research directions from information-theory-based modeling, which may lead to promising new approaches to advancing the field.

show abstract

“…Since then, it was applied to derive several clustering algorithms for a wide variety of applications such as: text classification [8], galaxy spectra classification [9], speaker recognition [10], among others. Further information-theoretic extensions of the IB were recently considered in [11]- [15]. In particular, in some of these works it is shown that the same characterization of the rate and distortion tradeoff is obtained when the distortion metric is not necessarily additive as assumed in (2).…”

Section: A Related Workmentioning

confidence: 99%

Compression-Based Regularization With an Application to Multitask Learning

Vera

Vega

Piantanida

2018

IEEE J. Sel. Top. Signal Process.

Self Cite

View full text Add to dashboard Cite

This paper investigates, from information theoretic grounds, a learning problem based on the principle that any regularity in a given dataset can be exploited to extract compact features from data, i.e., using fewer bits than needed to fully describe the data itself, in order to build meaningful representations of a relevant content (multiple labels). We begin by introducing the noisy lossy source coding paradigm with the log-loss fidelity criterion which provides the fundamental tradeoffs between the cross-entropy loss (average risk) and the information rate of the features (model complexity). Our approach allows an information theoretic formulation of the multi-task learning (MTL) problem which is a supervised learning framework in which the prediction models for several related tasks are learned jointly from common representations to achieve better generalization performance. Then, we present an iterative algorithm for computing the optimal tradeoffs and its global convergence is proven provided that some conditions hold. An important property of this algorithm is that it provides a natural safeguard against overfitting, because it minimizes the average risk taking into account a penalization induced by the model complexity. Remarkably, empirical results illustrate that there exists an optimal information rate minimizing the excess risk which depends on the nature and the amount of available training data. An application to hierarchical text categorization is also investigated, extending previous works.

show abstract

The multi-layer information bottleneck problem

Cited by 10 publications

References 13 publications

The Double-Sided Information-Bottleneck Function

The Double-Sided Information-Bottleneck Function

A Brief Survey of Telerobotic Time Delay Mitigation

Compression-Based Regularization With an Application to Multitask Learning

Contact Info

Product

Resources

About