Small urban green spaces are abundant in densely populated urban areas, but little is known about their impact on the urban heat island effect and thermal comfort. Therefore, this study selected as research sites four small urban green spaces in a typical high-density built-up area, Chuo Ward in Tokyo, Japan. The ENVI-met software 5.1.1 simulation method was used to analyze these sites’ microclimate and thermal comfort conditions. The following are the results: (1) Small urban green spaces significantly reduce urban air temperatures, particularly during hot weather, with temperature reductions ranging from 2.40 °C to 2.67 °C, consistently lower than the highest temperatures in Tokyo’s Chuo Ward, mainly between 1:00 and 2:00 p.m. (2) Thermal comfort analysis indicates that small urban green spaces can significantly improve urban thermal comfort during the day, particularly around noon, by reducing one or two thermal comfort levels compared to typical urban street blocks. However, these differences gradually diminish throughout the evening and night, and thermal comfort inside and outside green spaces becomes more uniform. (3) Green space size is not the only factor influencing thermal comfort; the layout of plants within the green space and the layout of the surrounding buildings also have an impact. Despite their small size, even small green spaces can significantly enhance comfort. This study highlights the need to promote urban sustainability through the extensive integration of small green spaces in dense urban environments. Small green spaces can serve as a high-frequency, low-cost solution for environmental sustainability by addressing the increasingly severe urban heat island effect as well as environmental challenges that in the urbanization process.