A human being comes into contact with the environment through the senses. That is why in the space of cities, where various intense stimuli negatively affect the living of people, there is important greenery that has a positive impact. Significant types of gardens within urban green areas are sensory gardens. In our article, we intended to answer the question of what specifically affects the formation of deeper human–garden relations in urban, publicly accessible gardens designed to have a sensory impact. Our research was conducted mainly in Poland. We used a method of assessing the behavior of garden visitors, using a five-point scale. We found that the existence of specific interiors in gardens that have been designed in such a way as to stimulate two to three selected senses, which we call the leading senses, can create an environment that allows for deeper relationships with the garden. We also concluded that when designing a public sensory garden, adaptation to specific user groups is one of the most important guidelines. A deeper contact with the sensory garden for people visiting a city, e.g., tourists, may occur especially when there is a positive surprise or when an additional need of this group is met in the garden. Sensory gardens, although they are a relatively new type of urban greenery, can become a permanent element of cities if they are carefully designed and meet the expectations of their recipients.