The article is devoted to the current topic of analysis of mathematical models for countering cyber fraud in banks. This problem is due to the security risks growth in the banking system, which are formed by fraudsters' cyberattacks and cybercrimes implementation. Therefore, the priority task for cyberbanking security is the application of modern mathematical methods to analyse the sources of cyber attacks, identify threats and losses in the banking services market, identify cyber-attacks and assess the scenario of potential cyber risk, etc. The article analyses the most widespread types of cyber fraud: social engineering, phishing, stalking, farming, DoS attacks, online fraud, potentially unwanted programs, etc. The study also considered a model of cognitive computing and detection of suspicious transactions in banking cyber-physical systems based on quantum computing in BCPS for the post-quantum era. The advantages, disadvantages and results of the model are defined. Predictive modelling is proposed to detect fraud in real-time by analysing incoming bank transactions with payment cards. Within the framework of this method, such models are used for the classification of fraud detection as logistic regression, a decision tree, and a narrower technique - a random forest decision tree. The study also considered using the harmonic search algorithm in neural networks to improve fraud detection in the banking system. It is found that although this model has the advantage of learning ability based on past behaviour, there are difficulties in the long-term processing of many neural networks. The stages of model implementation are also given. In addition, the modelling of credit card fraud detection is based on using two types of models: supervised and unsupervised. Supervised models include logistic regression, K-nearest neighbours, and extreme gradient boosting. The one-class support vector model, restricted Boltzmann model, and generative-competitive network are considered among uncontrolled generative models.
