Building extraction plays an important role in urban information analysis, which is helpful for urban planning and disaster monitoring. With the improvement of SAR resolution, rich detailed information in urban areas is revealed, but the discretized features also pose challenges for object detection. This paper addresses the problem of individual high-rise building extraction based on single high-resolution SAR image. Different from previous methods that require building facades to be presented in specific appearances, the proposed method is suitable for extraction of various types of high-rise buildings. After analyzing the SAR images of many types of high-rise buildings, we establish a unified high-rise building part model, on the basis of a scattering mechanism of building structures, to describe the facade characteristics of high-rise buildings, including a facade regularity part, facade bright line part, double bounce part, and their spatial topological relationships. A three-level high-rise building extraction framework, named HRBE-PM, is proposed based on the high-rise building part model. At the pixel level, a modified spot filter is used to extract bright spots and bright lines of different scales simultaneously to obtain salient features. At the structure level, building parts are generated based on the salient features according to semantic information. At the object level, spatial topological information between parts is introduced to generate building hypotheses. We define two indicators, completeness and compactness, to comprehensively evaluate each building hypothesis and select the optimal ones. After postprocessing, the final high-rise building extraction results are obtained. Experiments on two TerraSAR-X images show that the high-rise building extraction precision rate of the HRBE-PM method is above 85.29%, the recall rate is above 82.95%, and the F1-score is above 0.87. The results indicate that the HRBE-PM method can accurately extract individual high-rise buildings higher than 24 m in most dense scenes, regardless of building types.