Identifying Latent Structures in Panel Data

Abstract

This paper provides a novel mechanism for identifying and estimating latent group structures in panel data using penalized techniques. We consider both linear and nonlinear models where the regression coefficients are heterogeneous across groups but homogeneous within a group and the group membership is unknown. Two approaches are considered -penalized profile likelihood (PPL) estimation for the general nonlinear models without endogenous regressors, and penalized GMM (PGMM) estimation for linear models with endogeneity. In both cases we develop a new variant of Lasso called classifier-Lasso (C-Lasso) that serves to shrink individual coefficients to the unknown group-specific coefficients. CLasso achieves simultaneous classification and consistent estimation in a single step and the classification exhibits the desirable property of uniform consistency. For PPL estimation C-Lasso also achieves the oracle property so that group-specific parameter estimators are asymptotically equivalent to infeasible estimators that use individual group identity information. For PGMM estimation the oracle property of C-Lasso is preserved in some special cases. Simulations demonstrate good finite-sample performance of the approach both in classification and estimation. Empirical applications to both linear and nonlinear models are presented.JEL Classification: C33, C36, C38, C51

Homogeneity pursuit in panel data models: Theory and application

“…This paper follows earlier work by Su, Shi, and Phillips (2016) by studying a linear panel data model with latent structures that embody unknown homogeneous elements. It is assumed that the cross-sectional units can be classified into a small number of groups with homogeneous slopes within each group and heterogeneity across groups.…”

“…In addition, machine learning methods are also used to extract group patterns by using penalized extremum estimation. In particular, Su et al (2016) developed classifier-Lasso (C-Lasso) in which the penalty takes an additive-multiplicative form that forces the parameters to form into different groups. Coupled with the C-Lasso method, Su et al proposed a Bayesian-type information criterion to determine the number of groups.…”

“…Second, to use more information from the preliminary estimates, we extend the ordered segmentation concept proposed in Ke et al to the segmentation net, which enables us to extract groups more accurately. Just like CARDS for cross-section data or the Su et al (2016) C-Lasso for panel data, Panel-CARDS can identify the number of groups and estimate the parameters at the same time.…”

Determining the number of groups in latent panel structures with an application to income and democracy

“…Recently, latent group structures have received much attention in the panel data literature; see, for example, Sun (2005), Lin and Ng (2012), Deb and Trivedi (2013), Bonhomme and Manresa (2015;BM hereafter), Sarafidis and Weber (2015), Ando and Bai (2016), Bester and Hansen (2016), Su, Shi, and Phillips (2016;SSP hereafter), and Su and Ju (forthcoming). In comparison with some other popular approaches to model unobserved heterogeneity in panel data models such as random coefficient models (see, e.g., Hsiao (2014, Chapter 6)), one important advantage of the latent group structure is that it allows flexible forms of unobservable heterogeneity while remaining parsimonious.…”

“…

We consider a latent group panel structure as recently studied by Su, Shi, and Phillips (2016), where the number of groups is unknown and has to be determined empirically. We propose a testing procedure to determine the number of groups.

…”