The presence of small green spaces throughout high-density urban areas is essential for enhancing thermal comfort and combating the urban heat island effect. Extensive research has been conducted on plants, which play a key role in cooling green spaces. However, the majority of research focuses on trees, overlooking the efficiency of vertical plant configurations, especially shrubs. Hence, this paper selected a typical small green space in Tokyo’s high-density urban area and employed scenario simulation to examine how various vertical plant configurations impact the microclimate and thermal comfort of small green spaces, with a particular emphasis on quantifying the efficiency of shrubs. The research primarily discovered four findings. First, trees significantly outperform shrubs in improving thermal comfort, especially during periods of strong sunlight during the day, leading to a direct modification of the thermal comfort levels. Second, while shrubs (height < 2 m) usually do not make a substantial difference in the overall thermal comfort levels of small green spaces, they are effective in reducing areas classified as “hot” and “very hot” during the day, with their cooling effect becoming more pronounced in the late afternoon. Third, when small green spaces combine trees and shrubs in their plant configuration, taller shrubs (within 0–2 m) exhibit a more pronounced cooling effect and improvement in thermal comfort. The conclusions of this study could offer guidance for the planning, design, and renovation of small green spaces in high-density urban areas.