Mental health is one of the main factors that significantly affect one’s life. Previous studies suggest that streets are the main activity space for urban residents and have important impacts on human mental health. Existing studies, however, have not fully examined the relationships between streetscape characteristics and people’s mental health on a street level. This study thus aims to explore the spatial patterns of urban streetscape features and their associations with residents’ mental health by age and sex in Zhanjiang, China. Using Baidu Street View (BSV) images and deep learning, we extracted the Green View Index (GVI) and the street enclosure to represent two physical features of the streetscapes. Global Moran’s I and hotspot analysis methods were used to examine the spatial distributions of streetscape features. We find that both GVI and street enclosure tend to cluster, but show almost opposite spatial distributions. The Results of Pearson’s correlation analysis show that residents’ mental health does not correlate with GVI, but it has a significant positive correlation with the street enclosure, especially for men aged 31 to 70 and women over 70-year-old. These findings emphasize the important effects of streetscapes on human health and provide useful information for urban planning.