For the cultural heritage gardens in the urban environment, modern high-rise buildings inevitably change their original landscape and form a new landscape experience with visual impact. Whether cultural heritage gardens and modern cities can coexist harmoniously is one of the critical issues to achieve their sustainable development. This research aimed to find an indicator of landscape morphology, which can predict the visitor’s cognition for such cultural landscape forms. This study surveyed tourists’ preferences in six selected cultural heritage gardens in Tokyo. We used hemispheric panoramas to calculate the view factors of certain elements of the landscape at the observation points. The results showed that Sky View Factor was a positive predictor of tourists’ preference, and this predictability did not change significantly with the attributes of tourists. We also found that tourists’ attitudes towards the high-rise buildings outside the gardens have become more tolerant and diverse. These findings could be applied to predict visitors’ perception preference of cultural heritage landscape in the context of urban renewal, contributing to the sustainable development of cultural heritage landscape and urbanization.