Buildings are one of the most important goals of human transformation of the Earth’s surface. Therefore, building extraction (BE), such as in urban resource management and planning, is a task that is meaningful to actual production and life. Computational intelligence techniques based on convolutional neural networks (CNNs) and Transformers have begun to be of interest in BE, and have made some progress. However, the BE methods based on CNNs are limited by the difficulty in capturing global long-range relationships, while Transformer-based methods are often not detailed enough for pixel-level annotation tasks because they focus on global information. To conquer the limitations, a multi-scale Transformer (MSTrans) is proposed for BE from high-resolution remote sensing images. In the proposed MSTrans, we develop a plug-and-play multi-scale Transformer (MST) module based on atrous spatial pyramid pooling (ASPP). The MST module can effectively capture tokens of different scales through the Transformer encoder and Transformer decoder. This can enhance multi-scale feature extraction of buildings, thereby improving the BE performance. Experiments on three real and challenging BE datasets verify the effectiveness of the proposed MSTrans. While the proposed approach may not achieve the highest Precision and Recall accuracies compared with the seven benchmark methods, it improves the overall metrics F1 and mIoU by 0.4% and 1.67%, respectively.