Web Document Clustering with Multi-view Information Bottleneck

Gao, Yan; Gu, Shiwen; Xia, Liming; Yao-ping, Fei

doi:10.1109/cimca.2006.232

Cited by 5 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The general framework includes the IB and PF problems as special cases. We proposed solving the general problem with splitting methods that are capable of solving large-scale problems which connect to the recent trends on multi-view learning [46], [47] and multi-source privacy problems [47], [48], [49], [50], [51].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

A Linearly Convergent Douglas-Rachford Splitting Solver for Markovian Information-Theoretic Optimization Problems

Huang¹,

Gamal²,

Gamal³

2022

Preprint

View full text Add to dashboard Cite

In this work, we propose solving the Information bottleneck (IB) and Privacy Funnel (PF) problems with Douglas-Rachford Splitting methods (DRS). We study a general Markovian information-theoretic Lagrangian that formulates the IB and PF problems into a convex-weakly convex pair of functions in a unified framework. Exploiting the recent non-convex convergence analysis for splitting methods, we prove the linear convergence of the proposed algorithms using the Kurdyka-Łojasiewicz inequality. Moreover, our analysis is beyond IB and PF and applies to any convex-weakly convex pair objectives. Based on the results, we develop two types of IB solvers, with one improves the performance of convergence over existing solvers while the other is linearly convergent independent to the relevance-compression trade-off, and a class of PF solvers that can handle both random and deterministic mappings. Empirically, we evaluate the proposed DRS solvers for both the IB and PF problems with our gradientdescent-based implementation. For IB, the proposed solvers result in solutions that are comparable to those obtained through the Blahut-Arimoto-based benchmark and is convergent for a wider range of the penalty coefficient than existing solvers. For PF, our non-greedy solvers can explore the information plane better than the clustering-based greedy solvers. TABLE I SUMMARY OF THE CONVERGENCE ANALYSIS FOR TWO-BLOCK NON-CONVEX SPLITTING METHODS Reference Algorithm Convergence Conditions Rate of Conv. Properties of Functions Linear Constraints Ap − Bq2α)µ 2 B * locally linear F:convex G:σ G -weakly convex G is Lq-smooth A:full row rank B:positive definite Algorithm (7) 0 < α < 2 c > Mq[ ασ G Mq +φq 4−2α ] † locally linear F :convex G:σ G -weakly convex G:Mq-Lipschitz continuous G are Lq-smooth A:positive definite, B:full row rank Jia et al. [26] Prox. ADMM (α = 1) c > σ G + σ 2 G +8L 2 q 2µ 2 B locally linear F :convex G:σ G -weakly convex G is Lq-smooth A:positive definite B:positive definite Themelis et al. [19] 0 < α < 2; 2 ≤ α < 4 c > Lp; α/Lp−δp 4

show abstract

Section: Discussionmentioning

confidence: 99%

“…Following the steps ( 44), ( 46) and (48) in Appendix G, we start from (49). Define L k c := L c (p k , q k , ν k ) the function value evaluated with variables at step k for simplicity:…”

Section: Appendix I Proof Of Lemma 10mentioning

confidence: 99%

A Linearly Convergent Douglas-Rachford Splitting Solver for Markovian Information-Theoretic Optimization Problems

Huang¹,

Gamal²,

Gamal³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, combining all the observations in one giant view will result in an exponential increase in complexity (curse of dimensionality). A basic assumption in the multiview learning literature is the conditional independence [19,10], where the observations of all views {X (i) } V i=1 are independent given the target variable Y . That is, p({x}|y) = V i=1 p(x (i) |y).…”

Section: Multiview Information Bottleneckmentioning

confidence: 99%

“…This corresponds only to the compression part of the IB method and their goal was to maximize the marginal of a linear classifier and was limited to binary tasks. In [10], each view was treated as a single-view IB problem followed by a post-processing stage that aims to maximize the mutual information between the view-specific representations to encourage agreement of clustering hypotheses. More recently, [11] proposed a bottom-up, heuristic multiview IB objective which maximizes view-specific information, view-shared information, and inter-cluster correlation while compressing the observations simultaneously.…”

Section: Introductionmentioning

confidence: 99%

On the Multi-View Information Bottleneck Representation

Huang¹,

Gamal²,

Gamal³

2022

Preprint

View full text Add to dashboard Cite

In this work, we generalize the information bottleneck (IB [1]) approach to the multi-view learning context. The exponentially growing complexity of the optimal representation motivates the development of two novel formulations with more favorable performance-complexity tradeoffs. The first approach is based on forming a stochastic consensus and is suited for scenarios with significant representation overlap between the different views. The second method, relying on incremental updates, is tailored for the other extreme scenario with minimal representation overlap. In both cases, we extend our earlier work on the alternating directional methods of multiplier (ADMM [2]) solver and establish its convergence and scalability. Empirically, we find that the proposed methods outperform state-of-the-art approaches in multi-view classification problems under a broad range of modelling parameters.

show abstract