Unsupervised learning is an important problem in statistics and machine learning with a wide range of applications. In this paper, we study clustering of high-dimensional Gaussian mixtures and propose a procedure, called CHIME, that is based on the EM algorithm and a direct estimation method for the sparse discriminant vector. Both theoretical and numerical properties of CHIME are investigated. We establish the optimal rate of convergence for the excess mis-clustering error and show that CHIME is minimax rate optimal. In addition, the optimality of the proposed estimator of the discriminant vector is also established. Simulation studies show that CHIME outperforms the existing methods under a variety of settings. The proposed CHIME procedure is also illustrated in an analysis of a glioblastoma gene expression data set and shown to have superior performance.Clustering of Gaussian mixtures in the conventional low-dimensional setting is also considered. The technical tools developed for the highdimensional setting are used to establish the optimality of the clustering procedure that is based on the classical EM algorithm.

No abstract

Exactly what is meant by a ‘complex’ network is not clear; however, what is clear is that it is something other than a random graph. Complex networks arise in a wide range of real social, technological and physical systems. In all cases, the most basic categorization of these graphs is their node degree distribution. Particular groups of complex networks may exhibit additional interesting features, including the so-called small-world effect or being scale-free. There are many algorithms with which one may generate networks with particular degree distributions (perhaps the most famous of which is preferential attachment). In this paper, we address what it means to randomly choose a network from the class of networks with a particular degree distribution, and in doing so we show that the networks one gets from the preferential attachment process are actually highly pathological. Certain properties (including robustness and fragility) which have been attributed to the (scale-free) degree distribution are actually more intimately related to the preferential attachment growth mechanism. We focus here on scale-free networks with power-law degree sequences—but our methods and results are perfectly generic.

Summary The paper develops optimality theory for linear discriminant analysis in the high dimensional setting. A data‐driven and tuning‐free classification rule, which is based on an adaptive constrained l1‐minimization approach, is proposed and analysed. Minimax lower bounds are obtained and this classification rule is shown to be simultaneously rate optimal over a collection of parameter spaces. In addition, we consider classification with incomplete data under the missingness completely at random model. An adaptive classifier with theoretical guarantees is introduced and the optimal rate of convergence for high dimensional linear discriminant analysis under the missingness completely at random model is established. The technical analysis for the case of missing data is much more challenging than that for complete data. We establish a large deviation result for the generalized sample covariance matrix, which serves as a key technical tool and can be of independent interest. An application to lung cancer and leukaemia studies is also discussed.