As a code-domain non-orthogonal multiple access technique, sparse code multiple access (SCMA) is considered as a promising technique for future wireless Internet of Things (IoT) networks. The minimum Euclidean distance (MED) and minimum product distance (MPD) have been highlighted as the key performance indicators of the codebook in additive gaussian white noise (AWGN) and downlink Rayleigh channels respectively. In this paper, based on the mother codebook, a novel codebook design scheme is proposed to achieve better error performance in both AWGN and downlink Rayleigh channels. The problem of constructing the mother codebook is considered as the quadratic assignment problem (QAP), where the Tabu searching algorithm is employed to reduce the complexity of searching for the best permutation result. Then, the rotation matrix is adopted to find the best degrees for the generation of the constellation group, and two algorithms are proposed to assign the obtained constellation in factor graph matrix. Taking the degree optimization and the constellation assignments into joint consideration, an improved unified optimization is further explored to maximize the MED of each user. Besides, a novel polarized modulation scheme is proposed, which places the symbols in the three dimensional (3D) stokes parameters to improve the performance of the system. Finally, simulation results are provided to show the performance of the proposed codebooks, and the comparisons of symbol error performance (SER) in different codebooks are also discussed in detail.INDEX TERMS Sparse code multiple access (SCMA), codebook design, minimum Euclidean distance, minimum product distance, three dimensional SCMA codebooks