Due to the little variation in defect points, tile block defect detection typically detects subtle defects in large-format images, allowing defective characteristics to be displayed regionally. Traditional convolutional neural network architectures that extract regional features take into account the connection between regional features simply, resulting in the presence of region-specific bias, which makes tile block defect detection still a challenging task. To address this challenge, this paper divides feature information into patches that can represent different regional features. Additionally, the relationship between different patches and tile block defects is studied; as a result, this paper proposes a new attention mechanism called the Cross Patch Attention Module (CPAM). Since the regional performance of patches is consistent with the tile block defect characteristics, CPAM can distinguish various regional features by patches. Then, in order to create reliable one-dimensional patch information, CPAM provides a method to connect patches linearly in two spatial directions. This takes into account the correlation of adjacent patches in various spatial directions. Finally, by extracting the regional characteristics of patches, CPAM can successfully assist the model in distinguishing the importance of different patches. The experimental results demonstrate that CPAM has excellent performance for tile block defect detection, and plugging CPAM into different end-to-end models can have a good gain effect, which can effectively and stably help the model to complete the task of tile block defect detection.