Heritage sites are gradually becoming more vulnerable, due to natural decay and deterioration, effects of climate change, and non-climatic factors. An increased number of non-climatic human-induced impacts are causing significant problems in managing and conserving cultural heritages around the globe. For location and liberal cultural conceptualization, Bangladesh has a rich historical background with many remarkable cultural and natural heritage. Among many, the historic mosque city of Khalifatabaad, Bagerhat was enlisted as one of the world heritage sites by UNESCO in 1985 which is the only proven evidence of the ‘Khan-e-Jahan style’. UNESCO has already mentioned a few non-climatic factors which also made the heritage sites vulnerable along with the various climatic factors. Conservation of the historic landscape, buffer zone, and heritage property area is also addressed there. To identify the responsible human-induced non-climatic factors and their degree of impact on heritage sites are the primary aims of this research. Through site survey, detailed observation, documentation, oral statements records, and proper qualitative analysis the non-climatic issues like weak land-use policy, tourism policy & monitoring, implementation of land law and lack of social awareness are identified. For systematic data summary and findings, the Driver-pressure-state-impact-response (DPSIR) framework has been used.