Cross validation in LASSO and its acceleration

Obuchi, Tomoyuki; Kabashima, Yoshiyuki

doi:10.1088/1742-5468/2016/05/053304

Cited by 59 publications

(53 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We chose the largest value of λ for which the CV error was smaller than the sum of the minimum CV error and the standard error of CV errors. Because CV incurs high computational costs, we used the approximation of leave-one-out CV errors (Obuchi and Kabashima, 2016). A leave-one-out CV error and its standard error is approximated by the mean and the standard deviation of

{true(1 + \sum_{k, l} X_{j k}^{false(i false)} X_{j l}^{(i)} χ_{k l}^{(i j)} true)}^{2} {true({\hat{s}}_{j}^{(i)} - s_{j}^{(i)} true)}^{2},

over all samples j .…”

Section: Methodsmentioning

confidence: 99%

FIT: statistical modeling tool for transcriptome dynamics under fluctuating field conditions

Iwayama

Aisaka²,

Kutsuna

et al. 2017

Bioinformatics

View full text Add to dashboard Cite

MotivationConsiderable attention has been given to the quantification of environmental effects on organisms. In natural conditions, environmental factors are continuously changing in a complex manner. To reveal the effects of such environmental variations on organisms, transcriptome data in field environments have been collected and analyzed. Nagano et al. proposed a model that describes the relationship between transcriptomic variation and environmental conditions and demonstrated the capability to predict transcriptome variation in rice plants. However, the computational cost of parameter optimization has prevented its wide application.ResultsWe propose a new statistical model and efficient parameter optimization based on the previous study. We developed and released FIT, an R package that offers functions for parameter optimization and transcriptome prediction. The proposed method achieves comparable or better prediction performance within a shorter computational time than the previous method. The package will facilitate the study of the environmental effects on transcriptomic variation in field conditions.Availability and ImplementationFreely available from CRAN (https://cran.r-project.org/web/packages/FIT/).Supplementary information Supplementary data are available at Bioinformatics online

show abstract

{true(1 + \sum_{k, l} X_{j k}^{false(i false)} X_{j l}^{(i)} χ_{k l}^{(i j)} true)}^{2} {true({\hat{s}}_{j}^{(i)} - s_{j}^{(i)} true)}^{2},

over all samples j .…”

Section: Methodsmentioning

confidence: 99%

FIT: statistical modeling tool for transcriptome dynamics under fluctuating field conditions

Iwayama

Aisaka²,

Kutsuna

et al. 2017

Bioinformatics

View full text Add to dashboard Cite

show abstract

“…Here, some outliers exhibiting extraordinary small CV errors are omitted. At the CV error minimum, the solution with K = 10 is obtained, which is comparable with K = 9 of the LASSO solution at the minimum CV error [33,37]. In the case of LASSO, it is common to select a sparser solution than the one at the CV error minimum according to the one-standard error rule [10,40].…”

Section: A Real-world Dataset: Type Ia Supernovaementioning

confidence: 99%

“…Readers may doubt the effectiveness of this statement, because the input MSE ǫ x cannot be computed for realistic settings with unknown true signals. As explained ‡ In [33], essentially the same analysis is done for LASSO, but a wrong terminology is used. The quantity R is termed Youden's index in that study, but it is contradictory to the conventional terminology.…”

Section: Receiver Operating Characteristic Curvementioning

confidence: 99%

“…This is naturally understood from the scaling argument presented in sec. 5.1.1 in [33]. The corresponding computational time of the CD algorithm convergence and the approximate formula are given in the right panel.…”

Section: Accuracy Of the Approximate CV Formulamentioning

confidence: 99%

“…Our dataset is a part of the data from [35,36], which is screened by a certain criterion [37]. This dataset was treated by a number of sparse estimation techniques recently, and a set of important variables, which is known to be empirically significant, has been reproduced [33,37,38,39]. In those studies, the LASSO and ℓ 0 cases are treated, and the CV is employed for hyperparameter estimation.…”

Section: A Real-world Dataset: Type Ia Supernovaementioning

confidence: 99%

See 2 more Smart Citations

Cross validation in sparse linear regression with piecewise continuous nonconvex penalties and its acceleration

Obuchi

Sakata

2019

J. Phys. A: Math. Theor.

Self Cite

View full text Add to dashboard Cite

We investigate the signal reconstruction performance of sparse linear regression in the presence of noise when piecewise continuous nonconvex penalties are used. Among such penalties, we focus on the smoothly clipped absolute deviation (SCAD) penalty. The contributions of this study are three-fold: We first present a theoretical analysis of a typical reconstruction performance, using the replica method, under the assumption that each component of the design matrix is given as an independent and identically distributed (i.i.d.) Gaussian variable. This clarifies the superiority of the SCAD estimator compared with ℓ 1 in a wide parameter range, although the nonconvex nature of the penalty tends to lead to solution multiplicity in certain regions. This multiplicity is shown to be connected to replica symmetry breaking in the spin-glass theory, and associated phase diagrams are given. We also show that the global minimum of the mean square error between the estimator and the true signal is located in the replica symmetric phase. Second, we develop an approximate formula efficiently computing the cross-validation error without actually conducting the cross-validation, which is also applicable to the non-i.i.d. design matrices. It is shown that this formula is only applicable to the unique solution region and tends to be unstable in the multiple solution region. We implement instability detection procedures, which allows the approximate formula to stand alone and resultantly enables us to draw phase diagrams for any specific dataset. Third, we propose an annealing procedure, called nonconvexity annealing, to obtain the solution path efficiently. Numerical simulations are conducted on simulated datasets to examine these results to verify the consistency of the theoretical results and the efficiency of the approximate formula and nonconvexity annealing. The characteristic behaviour of the annealed solution in the multiple solution region is addressed. Another numerical experiment on a real-world dataset of Type Ia supernovae is conducted; its results are consistent with those of earlier studies using the ℓ 0 formulation. A MATLAB package of numerical codes implementing the estimation of the solution path using the annealing with respect to λ in conjunction with the approximate CV formula and the instability detection routine is distributed in [1].

show abstract

A Bayesian transfer sparse identification method for nonlinear ARX systems

Zhang,

Luan,

Ding

et al. 2024

Adaptive Control & Signal

View full text Add to dashboard Cite

SummaryIn this paper, we design a transfer sparse identification algorithm under the Bayesian framework through introducing other system knowledge into the system to be identified. This method provides a new identification solution for a nonlinear autoregressive model with exogenous inputs (NARX). The estimates of the transferred parameters are calculated by adding the transfer correction term to the un‐transferred estimates. To achieve this, a joint prior distribution is devised for the parameters, ultimately enhancing the efficient utilization of existing data, reducing the reliance on new data, and achieving more accurate identification. The maximized marginal likelihood method is used to find the transfer gain and the transfer information matrix in the transfer correction term. Meanwhile, in order to make the algorithm automatically adapt to different data, we design an automatic structure detection method based on the transfer framework. The method automatically determines the sparsity threshold based on the maximum inter‐class variance. Two examples are provided to demonstrate the advantages of our algorithm.

show abstract

Cross validation in LASSO and its acceleration

Cited by 59 publications

References 39 publications

FIT: statistical modeling tool for transcriptome dynamics under fluctuating field conditions

FIT: statistical modeling tool for transcriptome dynamics under fluctuating field conditions

Cross validation in sparse linear regression with piecewise continuous nonconvex penalties and its acceleration

A Bayesian transfer sparse identification method for nonlinear ARX systems

Contact Info

Product

Resources

About