The paper is devoted to the training of binary neural networks. They reduce the requirements for computing power and memory, which is especially important in conditions of limited resources. To date, binary networks do not provide sufficient recognition quality comparable to the quality of traditional floating-point networks, so the development of more efficient methods of training networks are highly relevant. In this paper, we propose a probabilistic model of a neural network that can be transformed into a binary network and consider a way of binarization. Experimental results have shown that our model with incremental binarization and subsequent fine-tuning makes it possible to achieve recognition accuracy of 97.5% for MNIST image classification problem when the accuracy of the binary model trained by Straight Through Estimation was 87.5%.