Commercial building is one of the major energy consumers that has drawn worldwide concerns. Heating, ventilating and air-conditioning (HVAC) system constitutes 40% of the total energy consumption in a typical commercial building. While the main objective of HVAC is to provide occupants with a comfort and safe environment, it currently lacks channels to recognize occupants' favourite temperatures as well as reflect their levels of comfort, e.g., too-cold or too-hot. Hence, it is hard to justify the energy consumption without considering end-user needs. Models of thermal comfort and predicted mean vote have been used to estimate such index, however, they are not widely adopted due to their complexity and inaccuracy. In this paper, we design the innovative system CarryEn, which first captures user's favourite temperature non-intrusively from their daily environment. We connect our system with the building management system (BMS), and optimize the setpoint temperature to occupants with our model. When the user moves into other rooms or buildings, his favourite setting will also be carried with him. Based on our experiments, CarryEn is able to achieve an improvement of 28.2% thermal satisfaction from occupants and save 13% of energy consumption.