Improved PAC-Bayesian Bounds for Linear Regression

Shalaeva, Vera; Esfahani, Alireza; Germain, Pascal; Petreczky, Mihály

doi:10.1609/aaai.v34i04.6020

Cited by 4 publications

(8 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, we make explicit all terms in the right-hand side. Section 5.1 extends our results to linear regression (which has been studied from the perspective of PAC-Bayes in the literature, most recently by [15]). We also experimentally illustrate the behaviour of our bound.…”

Section: Introductionsupporting

confidence: 52%

PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses

Maxime

Guedj

Rivasplata

et al. 2021

Entropy

View full text Add to dashboard Cite

We present new PAC-Bayesian generalisation bounds for learning problems with unbounded loss functions. This extends the relevance and applicability of the PAC-Bayes learning framework, where most of the existing literature focuses on supervised learning problems with a bounded loss function (typically assumed to take values in the interval [0;1]). In order to relax this classical assumption, we propose to allow the range of the loss to depend on each predictor. This relaxation is captured by our new notion of HYPothesis-dependent rangE (HYPE). Based on this, we derive a novel PAC-Bayesian generalisation bound for unbounded loss functions, and we instantiate it on a linear regression problem. To make our theory usable by the largest audience possible, we include discussions on actual computation, practicality and limitations of our assumptions.

show abstract

Section: Introductionsupporting

confidence: 52%

PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses

Maxime

Guedj

Rivasplata

et al. 2021

Entropy

View full text Add to dashboard Cite

show abstract

“…Proof Theorem III.1. The proof follows the same lines as that of [17,Theorem 2], for the sake of completeness we repeat the basic steps. From Theorem II.1 it follows that (11) holds with probability at least 1 − δ.…”

Section: Systemsmentioning

confidence: 98%

“…In [7] PAC-Bayesian bounds for linear regression with a quadratic loss function was developed, later this bound was improved and extended to non i.i.d. data in [17]. In particular, in [17] the derived PAC-Bayesian error bound for non i.i.d linear regression problem was applied to learning ARX models.…”

Section: Introductionmentioning

confidence: 99%

“…data in [17]. In particular, in [17] the derived PAC-Bayesian error bound for non i.i.d linear regression problem was applied to learning ARX models. Note that in [17] control inputs were allowed, while in the current paper we consider only autonomous systems.…”

Section: Introductionmentioning

confidence: 99%

“…In particular, in [17] the derived PAC-Bayesian error bound for non i.i.d linear regression problem was applied to learning ARX models. Note that in [17] control inputs were allowed, while in the current paper we consider only autonomous systems. Since ARX models without inputs correspond to AR models, and the latter class represents a special case of autonomous stochastic LTI state-space representations, the present paper can be viewed as an extension of [17] to stochastic LTI statespace representations.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

PAC-Bayesian theory for stochastic LTI systems

Eringis¹,

Leth²,

Wiśniewski³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper we derive a PAC-Bayesian error bound for autonomous stochastic LTI state-space models. The motivation for deriving such error bounds is that they will allow deriving similar error bounds for more general dynamical systems, including recurrent neural networks. In turn, PAC-Bayesian error bounds are known to be useful for analyzing machine learning algorithms and for deriving new ones.

show abstract

PAC-Bayesian theory for stochastic LTI systems

Eringis

Leth

Wiśniewski

et al. 2021

2021 60th IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

In this paper we derive a Probably Approxilmately Correct(PAC)-Bayesian error bound for linear time-invariant (LTI) stochastic dynamical systems with inputs. Such bounds are widespread in machine learning, and they are useful for characterizing the predictive power of models learned from finitely many data points. In particular, with the bound derived in this paper relates future average prediction errors with the prediction error generated by the model on the data used for learning. In turn, this allows us to provide finite-sample error bounds for a wide class of learning/system identification algorithms. Furthermore, as LTI systems are a sub-class of recurrent neural networks (RNNs), these error bounds could be a first step towards PAC-Bayesian bounds for RNNs.

show abstract

Improved PAC-Bayesian Bounds for Linear Regression

Cited by 4 publications

References 7 publications

PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses

PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses

PAC-Bayesian theory for stochastic LTI systems

PAC-Bayesian theory for stochastic LTI systems

Contact Info

Product

Resources

About