To reduce the inpatient mortality and improve the quality of hospital management, we explore the relationship between temperatures and in-hospital mortality in a large sample across 10 years in Nanjing, Jiangsu. We collected 10 years’ data on patient deaths from a large research hospital. Distributed lag non-linear model (DLNM) was used to find the association between daily mean temperatures and in-hospital mortality. A total of 6160 in-hospital deaths were documented. Overall, peak RR appeared at 8 °C, with the range of 1 to 20 °C having a significantly high mortality risk. In the elderly (age ≥ 65 years), peak RR appeared at 5 °C, with range − 3 to 21 °C having a significantly high mortality risk. In males, peak RR appeared at 8 °C, with the range 0 to 24 °C having a significantly high mortality risk. Moderate cold (define as 2.5th percentile of daily mean temperatures to the MT), not extreme temperatures (≤ 2.5th percentile or ≥ 97.5th percentile of daily mean temperatures), increased the risk of death in hospital patients, especially in elderly and male in-hospital patients.