In this paper, a design and implementation of fixed-complexity sphere decoder (FSD) combined with interference mitigation algorithm for downlink (DL) multiuser (MU) multiple-input multipleoutput (MIMO) system is presented. To overcome performance degradation from inter-user interference in DL MU-MIMO systems, a hardware friendly optimized interference mitigation algorithm is proposed. The proposed algorithm achieves 0.5 dB near-optimal performance, which is 4 dB better than existing interference whitening filter for the soft-output decision. In this paper, both hard and soft-output FSD detectors are implemented with the proposed interference mitigation algorithm to function 4 × 4 antenna MIMO detection and interference mitigation for 4 additional streams. A fully pipelined architecture is employed to support 3.6 Gbps at 150 MHz for signals modulated by 64-QAM. From the synthesis result, the designed symbol detector has a gate count of 887 K for the FSD without pre-processor and a gate count of 1220 K for interference mitigation. By comparing previously implemented MIMO detectors, the proposed design represents a promising architecture for DL MU-MIMO systems in terms of significant performance improvement relative to inter-user interference and the compatible hardware implementation maintaining high-throughput within acceptable gate count.INDEX TERMS fixed-complexity sphere decoder (FSD), inter-user interference, MIMO detection, multiuser multi-input multi-output (MU-MIMO), VLSI design.