The spatial scaling of patterns and processes is a hot topic of research in landscape ecology, and different scales may yield completely inconsistent results. Therefore, to understand the impact of the scale effect on urban heat island effect, this study analyzes the correlation between surface temperature and landscape index at different spatial scales over Nanjing. The scale effect is calculated thorough curve fitting of the Pearson’s correlation coefficient between ten landscape indices and land surface temperature at different window sizes, and the optimal one is determined. We have found that landscape indices can be divided into exponential and Gaussian landscape indices whose correlation with land surface temperature at different windows conforms to binomial exponential or multi-Gaussian functions, respectively. The optimal window size is approximately 4000–5100 m for exponential landscape indices, 1000–2000 m for aggregation index (AI) and percentage of like adjacencies (PLADJ), 6330 m for contagion (CONTAG) and 4380 m for total edge contrast index (TECI). Moreover, CONTAG and TECI have a high correlation coefficient plateau where the Pearson correlation coefficient is high and changes by less than 0.03 as the window size changes by more than 3000 m, which makes it possible to decrease the window size in order to save the calculation time without an obvious decrease in the Pearson correlation coefficient. To achieve this, we proposed a suitable window selection function so that the window size becomes 4260 m and 2070 m, respectively. The window sizes obtained in this study are just suitable in Nanjing, but the window sizes in other cities can also be obtained by the method in this study. This study provides a reference for future research on the relationship between landscape pattern and land surface temperature and its driving mechanisms, as well as for the impact of urban land use planning on the heat island effect.