From a human-centric perspective, improving the spatial quality of historical districts is a key focus in achieving their refined governance. Based on a public survey performed in Zhangzhou Ancient City, we explored the differences in the perception of cultural landscape genes (CLGs) between residents and tourists with different individual characteristics, utilizing non-parametric tests and box-plot analysis methods. The findings indicate the following: Tourists have a stronger perception of CLGs related to site layout and architectural features than residents, while residents have a stronger perception of cultural customs. Gender-related differences in the perception of CLG are only significant among tourists, with females showing a higher level of CLG perception than males. Residents of different age groups show a “U-shaped” distribution in their perception of CLGs related to site layout, whereas senior tourists have a lower cognitive perception level. People with lower educational backgrounds among residents tend to perceive cultural customs more significantly, whereas tourists with diverse educational backgrounds do not perceive them as having a significant impact. Residence location, visit frequency, duration of residence, and residence nature are key positive factors affecting the perception of CLGs. Our study highlights the significance of considering the users’ characteristics in enhancing the refined governance of historical districts.