History and collective memory can be maintained through heritage building conservation because collective memory can be represented through an architectural object. It is important to preserve the whole story of the building, the architectural form, and the memory related to the building as well as the society so that all of those aspects can be passed down to the next generation. The conservation of both tangible and intangible characters of a heritage building is a key aspect to attain a sustainable city and society. When a heritage building is physically destroyed, the connection among people, the building, the collective memory of the building is also vanished and this condition can be considered as an example of terror in architecture. This paper discusses terror in architecture, especially when the building is slowly destroyed, and the memory that people have about the building is being attacked. The aim of the paper is to investigate whether there is any linkage between the physical changes of the building and collective memory of the society. This study appoints Pasar Johar Semarang as a study case since it is a heritage building that experienced some changes and misfortunes such as fire incident in 2015. In order to do so, this paper will connect some theories on heritage building conservation, collective memory, and terror in architecture. This paper concludes that Pasar Johar encounters terror in its architecture because the building is slowly damaged by physical intervention, ignorance from the authority and fire incident, at the end those situations cause the collective memory of society for the market slowly disappears.