Seismic fragility analysis is a crucial tool for assessing the seismic performance of buildings. In areas with dense clusters of tall buildings, the significant site-city interaction (SCI) effect alters wave propagation mechanisms, influencing the seismic fragility of structures. However, utilizing the incremental dynamic analysis method directly to calculate the seismic fragility of buildings considering the SCI effect encounters two main challenges. Firstly, it is unclear how many seismic scenarios need to be analyzed to fully capture the distribution of the SCI effect on building responses. Secondly, reducing the computational workload arising from the SCI effect and incremental dynamic analysis is a pressing concern. To address these challenges, this study examines the Shanghai Central Business District as a case study and employs a nonlinear numerical coupling scheme to analyze the SCI effect on building responses under different seismic wave inputs. The study investigates the minimum number of selected input motions required to achieve different accuracy levels. According to the findings, a minimum of seven pairs of seismic wave inputs are suggested for the SCI effect analysis. Moreover, the analysis demonstrates that low-rise buildings exhibit significantly larger variations in SCI-induced response changes compared to tall buildings. Therefore, a seismic fragility analysis method considering the SCI effect is proposed for low-rise buildings. This method offers a substantial reduction in computational workload while accurately capturing the SCI effect on building responses.