Owing to the nonstationary feature of heart sound signal which is often disturbed by noise, a method based on permutation entropy and factorization network in the classification of heart sound of congenital heart disease is proposed. Firstly, the Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) of normal and abnormal heart sound signals is performed to obtain several intrinsic mode function (IMF) components. Secondly, the optimal parameters in the permutation entropy algorithm are determined by C_C algorithm, and then according to the algorithm, the entropy values of the IMF components of each order are calculated to form high-dimensional eigenvectors. Finally, the recommended models of factorization machines (FMs) are used to classify the heart sounds and the generalisation performance of the FM network model are evaluated from accuracy and Area Under ROC Curve (AUC). The results show that the FM accuracy is 0.865, the AUC value is 0.887; the SVM accuracy is 0.812, and the AUC value is 0.805.