Most damage detection methods developed in the literature cannot give the locations and extent of damages under the presence of varying temperature condition. This is because temperature condition changes the vibration properties of a structure, which are commonly analyzed for damage detection, and temperature gradient throughout the structure makes it difficult to create a baseline for the undamaged structure, as the baseline is generally constructed using features obtained under a wide range of temperature conditions. In this paper, a new insight on how to approach damage detection using only a single temperature condition to create the baseline is proposed. This approach solves the damage detection under changing temperature problem in two stages by first quantifying the change of stiffness of all the elements in a structure due to temperature and damage effects, followed by removing the temperature effect, a global effect, to give the actual damage locations and extent. Using single temperature condition allows new measurements to be compared to a benchmark so that local deviation can be obtained, thus making the damaged elements identifiable. The proposed approach is tested using a beam structure model and a shear building under different gradient temperature conditions, and the results demonstrate that the method successfully eliminates the change in elemental stiffness due to temperature effect and gives correct damage locations and extent. The approach can be implemented with other existing damage detection methods that did not consider the effect of temperature so that structures under varying temperature condition can be analyzed.