Millimeter Wave (mmWave) massive multiple-input multiple-output (MIMO) systems with hybrid analog-digital architectures can greatly increase system capacity and communicate with multiple users at the same time. Accurate channel estimation is crucial for multi-user communications, but its accuracy is limited as the number of antennas and users increases. In the matrix high-dimensional operation for multi-user channel estimation, not only is it computationally intensive, but also the estimation accuracy is low. It makes our work turn to channel estimation of the user group within a certain region to improve the accuracy of estimation. In this paper, we propose a tensor dictionary manifold learning method for channel estimation and interference elimination of the multi-user mmWave massive MIMO system. A multi-user digital-analog mixed received signal model is presented. The tensor dictionary manifold learning scheme is proposed to model the received signal as a third-order low-rank tensor to handle the high-dimensional user, antenna, and channel. After segmentation, clustering and manifold learning, multiple tensor dictionary manifold models containing a group of user signals are fitted. Tensor dictionary manifold learning can take advantage of the inherent multi-domain properties of signals in the frequency, time, code and spatial domains to maintain inter-user correlation within a user group while reducing the high-dimensional channels of the user group. Using the convex relaxation property of the tensor alternating direction method, we propose a strategy to eliminate interference from other groups. And with the help of the multi-signal classification method, the channel parameters of user groups are obtained to improve the accuracy of multiuser channel estimation. This method can perform channel estimation for multiple users with only a few pilots, and improve the performance of the system. Numerical results confirm the good performance of this method.INDEX TERMS MIMO; channel estimation; tensor; dictionary learning; manifold; ADMM; MUSIC.
I. INTRODUCTIONMassive multiple-input multiple-output (MIMO) is the key technology of the future 5G wireless communication system [1]. It configures a large number of antenna elements in a small space to obtain large multiplexing gain [2] and improve the capacity of wireless communication [3]. Therefore, the massive MIMO system supports communication with multiple users. However, the communication between multiple users needs to ensure strict service quality and requires advanced signal processing techniques [4], [5], such as channel estimation, channel equalization and channel coding. Channel