In line with the concept of brand association, a famous Designation of Origin (DO) brand image may influence the non-visitor perception of the destination image, but there is little research on this topic. This study investigates the effect of the DO brand image on destination image and non-visitor visiting intention toward a tea culture tourism destination. A systematic survey was conducted in Guangdong, China. The results of 311 valid samples show that all four dimensions (feature, value, functional, and affective) of the DO brand image have a significant positive impact on destination image, but only two cognitive dimensions (feature and functional) of the DO brand image directly influence visiting intention. Therefore, this study confirms that destination image mediates the relationship between DO brand image and tourists’ visit intention. This study contributes to destination image research by linking DO brand and destination image to promote a tourist destination. It also provides implications to tea culture tourism research by integrating a measurement scale of tea image and recommending the practices of tea culture tourism.