Selection and Fusion of Categorical Predictors with L0-Type Penalties

Abstract

In regression modelling, categorical covariates have to be coded. Depending on the number of categorical covariates and on the number of levels they have, the number of coefficients can become huge. To reduce the model complexity, coefficients of similar categories should be fused and coefficients of non-influential categories should be set to zero. To this end, Lasso-type penalties on the differences of coefficients are a standard approach. However, the clustering/selection performance of this approach is sometimes poor–especially when the adaptive weights are badly conditioned or not existing. In some situations, there is no incentive to cluster similar categories. To overcome this, a [Formula: see text] penalty on the differences of coefficients is proposed, whereby the [Formula: see text] ‘norm’ is defined as the number of non-zero entries in a vector. The proposed penalty favours to find clusters of categories that share the same effect on the response variable while the estimation accuracy is comparable to Lasso-type penalties. Numerical experiments within the framework of generalized linear models are promising. For illustration, data on the unemployment rates in Germany is analyzed.

Modelling High-Dimensional Categorical Data using Nonconvex Fusion Penalties

“…While this shrinks the largest and smallest of the estimated coefficients together, the remaining coefficients lying in the open interval between these are unpenalised and so no grouping of the estimates is encouraged, as we observe in Figure 2; see also Oelker et al (2015) for a discussion of this issue in the context of ordinal variables.…”

“…the absolute value | jk + jl | by (| jk + jl | ) where ρ is a concave and non-decreasing penalty function (Ma & Huang, 2017;Oelker et al, 2015). However, this does not address the basic issue of a preference for groups of unequal sizes.…”

Modelling High-Dimensional Categorical Data Using Nonconvex Fusion Penalties

“…Whilst this shrinks the largest and smallest of the estimated coefficients together, the remaining coefficients lying in the open interval between these are unpenalised and so no grouping of the estimates is encouraged, as we observe in Figure 1; see also Oelker et al [2015] for a discussion of this issue in the context of ordinal variables.…”

Robust Haebara Linking for Many Groups: Performance in the Case of Uniform DIF

“…The implementation of robust Haebara linking in the sirt [36] package specifies a sequence of decreasing values of ε in the optimization, each using the previous solution as initial values (see [37] for a similar approach). It should be noted that alternative differentiable approximating function for the loss function ρ(x) = |x| p for values p nearby zero have been employed [38].…”