With the development of effective deep learning algorithms, it became possible to achieve high accuracy when conducting remote sensing analyses on very high-resolution images (VHRS), especially in the context of building detection and classification. In this article, in order to improve the accuracy of building detection and classification, we propose a Faster Edge Region Convolutional Neural Networks (FER-CNN) algorithm. This proposed algorithm is trained and evaluated on different datasets. In addition, we propose a new method to improve the detection of the boundaries of detected buildings. The results of our algorithm are compared with those of other methods, such as classical Faster Region Convolution Neural Network (Faster R-CNN) with the original VGG16 and the Single-Shot Multibox Detector (SSD). The experimental results show that our methods make it possible to obtain an average detection accuracy of 97.5% with a false positive classification rate of 8.4%. An additional advantage of our method is better resistance to shadows, which is a very common issue for satellite images of urban areas. Future research will include designing and training the neural network to detect small buildings, as well as irregularly shaped buildings that are partially obscured by shadows or other occlusions.