Design of full-duplex (FD) wireless systems faces many challenges, including selfinterference cancellation (SIC), capability to provide high capacity, high flexibility for operation, best usage of resources, etc. In this paper, we propose and investigate a multicarrier-division duplex (MDD) based hybrid beamforming system operated in FD mode, which is endowed with the advantages of both time-division duplex and frequency-division duplex. It also shares some merits of FD and allows to be free of self-interference (SI) in digital domain, but faces the same challenge of SI as the FD in analog domain. Hence in this paper, we first propose an adaptive beamforming assisted SI cancellation scheme with taking into account the practical requirement of analog-to-digital conversion (ADC). It can be shown that the proposed approach is capable of jointly coping with the desired signals' transmission and SI suppression. Then, channel estimation (CEst) in MDD/MU-MIMO system is proposed by exploiting the reciprocity between the uplink and downlink subcarrier channels that is provided by MDD. Correspondingly, the orthogonality-achieving pilot symbols are designed, and the least-square (LS) CEst as well as linear minimum mean-square error (LMMSE) CEst are derived. Finally, the performance of MDD/MU-MIMO systems employing the proposed SIC method is investigated, with respect to the SI cancellation capability, sum-rate potential, CEst performance, and the effect of CEst on the achievable performance. Our studies show that MDD/MU-MIMO provides an effective option for design of future wireless transceivers. INDEX TERMS Full-duplex, multicarrier-division duplex, self-interference cancellation, hybrid beamforming, channel estimation, least square, linear minimum mean square error.