Building pattern recognition is fundamental to a wide range of downstream applications, such as urban landscape evaluation, social analyses, and map generalization. Although many studies have been conducted, there is still a lack of satisfactory results, due to the imprecision of the relative direction model of any two adjacent buildings and the ineffective extraction methods. This study aims to provide an alternative for quantifying the direction and the spatial continuity of any two buildings on the basis of the Delaunay triangulation for the recognition of linear building patterns. First, constrained Delaunay triangulations (CDTs) are created for all buildings within each block and every two adjacent buildings. Then, the spatial continuity index (SCI), the direction index (DI), and other spatial relations (e.g., distance) of every two adjacent buildings are derived using the CDT. Finally, the building block is modelled as a graph based on derived matrices, and a graph segmentation approach is proposed to extract linear building patterns. In the segmentation process, the edges of the graph are removed first, according to the global thresholds of the SCI and distance, and are subsequently subdivided into subgraphs on direction rules. The proposed method is tested using three datasets. The experimental results suggest that the proposed method can recognize both collinear and curvilinear building patterns, given that the correctness values are all above 92% for the three study areas. The results also demonstrate that the novel SCI can effectively filter many insignificant neighbor relationships in the graph segmentation process. It is noteworthy that the proposed DI is capable of measuring building relative directions accurately and works efficiently in linear building pattern extraction.