Ising Model Selection Using $\ell_{1}$-Regularized Linear Regression: A Statistical Mechanics Analysis

Abstract

We theoretically investigate the performance of 1 -regularized linear regression ( 1 -LinR) for the problem of Ising model selection using the replica method from statistical mechanics. The regular random graph is considered under paramagnetic assumption. Our results show that despite model misspecification, the 1 -LinR estimator can successfully recover the graph structure of the Ising model with N variables using M = O (log N ) samples, which is of the same order as that of 1 -regularized logistic regression. Moreover, we provide a computationally efficient method to accurately predict the non-asymptotic performance of the 1 -LinR estimator with moderate M and N . Simulations show an excellent agreement between theoretical predictions and experimental results, which supports our findings.

On Model Selection Consistency of Lasso for High-Dimensional Ising Models

“…When the same assumptions are directly imposed on the sample covariance matrices, only Ω(d 2 log p) samples suffice. Compared to previous works [Lokhov et al, 2018, Meng et al, 2021, we provide a rigorous analysis of Lasso for Ising model selection and explicitly prove its consistency in the whole paramagnetic phase. Given the wide popularity and efficiency of Lasso, such rigorous analysis provides a theoretical backing for its practical use in Ising model selection, which can be viewed as a complement to that for Gaussian graphical models [Meinshausen et al, 2006, Zhao andYu, 2006].…”

“…To the best of our knowledge, this problem has not been rigorously studied, though there are several recent non-rigorous works in the statistical physics literature [Abbara et al, 2020, Meng et al, 2020. In particular, it is shown in Meng et al [2021] that even in the case with a misspecified quadratic loss, the neighborhood-based least absolute shrinkage and selection operator (Lasso) [Tibshirani, 1996] (termed as 1 -regularized linear regression ( 1 -LinR) in Meng et al [2021]) has the same order of sample complexity as 1 -LogR for random regular (RR) graphs in the whole paramagnetic phase [Mezard and Montanari, 2009]. Furthermore, Meng et al [2021] provides an accurate estimate of the typical sample complexity as well as a precise prediction of the non-asymptotic learning performance.…”

“…Furthermore, Meng et al [2021] provides an accurate estimate of the typical sample complexity as well as a precise prediction of the non-asymptotic learning performance. However, there are several limitations in Meng et al [2021]. First, since the replica method [Mezard and Montanari, 2009, Mézard et al, 1987, Nishimori, 2001, Opper and Saad, 2001 they use is a non-rigorous method from statistical mechanics, a rigorous mathematical proof is lacking.…”

“…First, since the replica method [Mezard and Montanari, 2009, Mézard et al, 1987, Nishimori, 2001, Opper and Saad, 2001 they use is a non-rigorous method from statistical mechanics, a rigorous mathematical proof is lacking. Second, the results in Meng et al [2021] are restricted to the ensemble of RR graphs since the obtained equations of state (EOS) explicitly depend on the eigenvalue distribution (EVD) of the covariance matrix, which is intractable for general graphs. In addition, since their analysis relies on the self averaging property [Mezard andMontanari, 2009, Nishimori, 2001], the results in Meng et al [2021] are meaningful in terms of the "typical case" rather than the worst case.…”

“…Second, the results in Meng et al [2021] are restricted to the ensemble of RR graphs since the obtained equations of state (EOS) explicitly depend on the eigenvalue distribution (EVD) of the covariance matrix, which is intractable for general graphs. In addition, since their analysis relies on the self averaging property [Mezard andMontanari, 2009, Nishimori, 2001], the results in Meng et al [2021] are meaningful in terms of the "typical case" rather than the worst case. Another closely related work is Lokhov et al [2018], which states that at high temperatures when the magnitudes of the couplings are approaching zero, both logistic and IOS losses can be approximated by a quadratic loss using the second-order Taylor expansion, so that both the 1 -LogR and RISE estimators behave similarly as the Lasso estimator at high temperatures.…”