Distributed Sufficient Dimension Reduction for Heterogeneous Massive Data

Abstract

We propose distributed sufficient dimension reduction to process massive data characterized with high dimensionality, huge sample size, and heterogeneity. To address the high dimensionality issue, we replace high dimensional explanatory variables with a small number of linear projections that are sufficient to explain the variabilities of the response variable. We allow for distinctive function maps for data scattered at different locations, which addresses the heterogeneity issue. We assume the dimension reduction subspaces at different local nodes are identical. This allows us to aggregate local results obtained from each local node to yield a final estimate on the center server. Sliced inverse regression and cumulative slicing estimation are explicitly studied. We investigate the non-asymptotic error bounds of the resulting dimensionality reduction. These theoretical results are further supported with simulation studies and an application to meta-genome data in American Gut Project.