This paper reviews the advances, challenges, and approaches of artificial intelligence (AI) and machine learning (ML) in the banking sector. The use of these technologies is accelerating in various industries, including banking. However, the literature on banking is scattered, making a global understanding difficult. This study reviewed the main approaches in terms of applications and algorithmic models, as well as the benefits and challenges associated with their implementation in banking, in addition to a bibliometric analysis of variables related to the distribution of publications and the most productive countries, as well as an analysis of the co-occurrence and dynamics of keywords. Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) framework, forty articles were selected for review. The results indicate that these technologies are used in the banking sector for customer segmentation, credit risk analysis, recommendation, and fraud detection. It should be noted that credit analysis and fraud detection are the most implemented areas, using algorithms such as random forests (RF), decision trees (DT), support vector machines (SVM), and logistic regression (LR), among others. In addition, their use brings significant benefits for decision-making and optimizing banking operations. However, the handling of substantial amounts of data with these technologies poses ethical challenges.