The transformation of the banking system is due to modern global trends and digitalization processes, which contribute to increasing the efficiency of the banking system operation and its competitiveness. The object of this study is the analysis of regulatory policy and justification of the foundations of institutional transformation of the banking sector in the conditions of digitalization. The conducted research made it possible to analyze the regulatory policy and establish the rationale for the institutional transformation of the banking sector in the conditions of digitalization in order to increase the efficiency of its activities and competitiveness, which means: putting the banking system on track to merge with the global financial system; transformation of the functioning of the National Bank of Ukraine in the direction of creating economically adaptable functions and, at the same time, implementing behavioral mechanisms of influence on banking institutions; implementation of methods of preventive rehabilitation of troubled banks; increasing the stabilization of the banking system due to strengthening the role of private banks; expansion of the functions of the Deposit Guarantee Fund of individuals, and; implementation of digital technologies in the activities of banking sector entities.