Regression from dependent observations

Daskalakis, Constantinos; Dikkala, Nishanth; Panageas, Ioannis

doi:10.1145/3313276.3316362

Cited by 16 publications

(37 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Corollary 5 of Theorem 1 quantifies that access to multiple samples typically decreases the reconstruction error by a factor of Ω( √ ℓ). As such, we get reconstruction guarantees which smoothly interpolate between the single-sample estimation setting considered by [9,14,21,30] and the more common 𝜔 (1)-sample estimation setting considered by [11,33,35,54,57]. Interestingly, instantiating our result to the latter setting we obtain guarantees which are competitive to that work, as shown in Corollary 6 and the middle row of Table 1.…”

Section: Introductionsupporting

confidence: 53%

“…Further, [21] studied linear regression with Ising model dependencies, which corresponds to learning 𝛽 together with multiple external field parameters. In comparison, Theorem 1 is the first to learn Ising models using one-sample from a complex family of matrices.…”

Section: Comparison To Prior Workmentioning

confidence: 99%

“…Prior to our work, the single-sample Ising model estimation literature had only studied quite restrictive special cases of our problem, namely the case 𝐽 * = 𝛽 𝐽 , where 𝐽 is a known matrix, and 𝛽 is an unknown scalar strength parameter [9,14], or slightly more general cases studied by follow-up work [9,21,30]. Restricted to this special case, our bounds provide quantitative improvements in the estimation error, as discussed in Section 2.4.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning Ising models from one or multiple samples

Dagan

Daskalakis

Dikkala

et al. 2021

Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing

Self Cite

View full text Add to dashboard Cite

There have been two main lines of work on estimating Ising models: (1) estimating them from multiple independent samples under minimal assumptions about the model's interaction matrix ; and (2) estimating them from one sample in restrictive settings. We propose a unified framework that smoothly interpolates between these two settings, enabling significantly richer estimation guarantees from one, a few, or many samples.Our main theorem provides guarantees for one-sample estimation, quantifying the estimation error in terms of the metric entropy of a family of interaction matrices. As corollaries of our main theorem, we derive bounds when the model's interaction matrix is a (sparse) linear combination of known matrices, or it belongs to a finite set, or to a high-dimensional manifold. In fact, our main result handles multiple independent samples by viewing them as one sample from a larger model, and can be used to derive estimation bounds that are qualitatively similar to those obtained in the afore-described multiple-sample literature. Our technical approach benefits from sparsifying a model's interaction network, conditioning on subsets of variables that make the dependencies in the resulting conditional distribution sufficiently weak. We use this sparsification technique to prove strong concentration and anti-concentration results for the Ising model, which we believe have applications beyond the scope of this paper. CCS CONCEPTS• Theory of computation → Sample complexity and generalization bounds; Models of learning; • Mathematics of computing → Markov networks.

show abstract

Section: Introductionsupporting

confidence: 53%

Section: Comparison To Prior Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Ising models from one or multiple samples

Dagan

Daskalakis

Dikkala

et al. 2021

Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…This approach was originally explored in the seminal paper of Chatterjee [21], where general conditions for √ Nconsistency of the MPL estimate for the model (2) were derived. 2 This result was subsequently extended to more general models in [7,31,32,36,60]. In particular, for model (3) Daskalakis et al [32] showed that given a single sample of observations (X i , Z i ) 1 i N from (3), the MPL estimate of the parameters (β, θ) is √ N -consistent, when the dimension d is fixed, under various regularity assumptions on the underlying network and the parameters.…”

Section: Introductionmentioning

confidence: 94%

High Dimensional Logistic Regression Under Network Dependence

Mukherjee¹,

Niu²,

Halder³

et al. 2021

Preprint

View full text Add to dashboard Cite

Logistic regression is one of the most fundamental methods for modeling the probability of a binary outcome based on a collection of covariates. However, the classical formulation of logistic regression relies on the independent sampling assumption, which is often violated when the outcomes interact through an underlying network structure, such as over a temporal/spatial domain or on a social network. This necessitates the development of models that can simultaneously handle both the network 'peer-effect' (arising from neighborhood interactions) and the effect of (possibly) high-dimensional covariates. In this paper, we develop a framework for incorporating such dependencies in a high-dimensional logistic regression model by introducing a quadratic interaction term, as in the Ising model, designed to capture the pairwise interactions from the underlying network. The resulting model can also be viewed as an Ising model, where the node-dependent external fields linearly encode the high-dimensional covariates. We propose a penalized maximum pseudo-likelihood method for estimating the network peer-effect and the effect of the covariates (the regression coefficients), which, in addition to handling the high-dimensionality of the parameters, conveniently avoids the computational intractability of the maximum likelihood approach. Consequently, our method is computational efficient and, under various standard regularity conditions, we show that the corresponding estimate attains the classical high-dimensional rate of consistency. In particular, our results imply that even under network dependence it is possible to consistently estimate the model parameters at the same rate as in classical (independent) logistic regression, when the true parameter is sparse and the underlying network is not too dense. As a consequence of the general results, we derive the rates of consistency of our proposed estimator for various natural graph ensembles, such as bounded degree graphs, sparse Erdős-Rényi random graphs, and stochastic block models.

show abstract

“…Even more recently, a line of study has investigated supervised learning problems under limited dependency between the samples (rather than the usual i.i.d. setting), using related techniques [DDP19,DDDJ19].…”

Section: Testing Problemmentioning

confidence: 99%

Testing Ising Models

Daskalakis

Dikkala

Kamath

2019

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

Given samples from an unknown multivariate distribution p, is it possible to distinguish whether p is the product of its marginals versus p being far from every product distribution? Similarly, is it possible to distinguish whether p equals a given distribution q versus p and q being far from each other? These problems of testing independence and goodness-of-fit have received enormous attention in statistics, information theory, and theoretical computer science, with sample-optimal algorithms known in several interesting regimes of parameters. Unfortunately, it has also been understood that these problems become intractable in large dimensions, necessitating exponential sample complexity.Motivated by the exponential lower bounds for general distributions as well as the ubiquity of Markov Random Fields (MRFs) in the modeling of high-dimensional distributions, we initiate the study of distribution testing on structured multivariate distributions, and in particular the prototypical example of MRFs: the Ising Model. We demonstrate that, in this structured setting, we can avoid the curse of dimensionality, obtaining sample and time efficient testers for independence and goodness-of-fit. One of the key technical challenges we face along the way is bounding the variance of functions of the Ising model.

show abstract

Regression from dependent observations

Cited by 16 publications

References 38 publications

Learning Ising models from one or multiple samples

Learning Ising models from one or multiple samples

High Dimensional Logistic Regression Under Network Dependence

Testing Ising Models

Contact Info

Product

Resources

About