In this paper, the reliability and security of ambient backscatter (AmBC) non-orthogonal multiple access (NOMA) systems on Nakagami-m distribution are studied. When an eavesdropper exists, the transmitter communicates with two NOMA users through the power domain multiplexing technology and successful interference cancellation (SIC) technology. Actually, assuming that in-phase and quadraturephase imbalance (IQI) is present in all nodes. To be specific, the precise expressions of the outage probability (OP) and intercept probability (IP) are derived. And Monte Carlo simulation coincides better with the theoretical images, which confirms the accuracy of theoretical derivation. A surprising discovery is that IQI diminishes the reliability of considered networks but strengthens confidentiality. Moreover, the decrease of the average signal-to-noise ratio (SNR) of the link from the transmitter to the eavesdropper can enhance the security. Furthermore, according to the back-propagation (BP) neural networks, we provide a forecast method to achieve real-time analysis of the security performance. Comparing to the popular machine learning algorithms, the results show that the prediction time is shorter and the prediction accuracy is averagely increased by 91.2%. Meanwhile, similar results can be obtained in the prediction of other communication models, which proves that the prediction algorithm has positive applicability.INDEX TERMS Physical layer security, non-orthogonal multiple access, in-phase and quadrature-phase imbalance, ambient backscatter communication, back-propagation neural network.