Commercial building is one of the major energy consumers worldwide. Among the building services, the heating, ventilating and air-conditioning (HVAC) system dominates the total energy consumption. Recent studies have proposed various approaches to audit, automate and optimize energy usage of the HVAC system. Nevertheless, these schemes seldom discuss human thermal comfort. To minimize complaints, the current practice of the facility management is to adopt very conservative temperatures, leading to massive waste of energy.In this paper, we actively take thermal comfort into consideration. We propose a participatory approach allowing the occupants provide feedback regarding their comfort levels. A major challenge for a participatory design is to reduce intrusiveness of the system. To this end, we develop a temperature comfort correlation model that can build a profile for each occupant. The decision of setpoint temperature can be primarily model-driven, requiring minimal inputs of the occupants. We validated our model with field experiments. Besides, we developed a setpoint optimization algorithm to handle the diverging thermal requirements of multiple occupants in same room, and examined the model with simulations. We implemented our design and conducted field experiments in a University and a commercial office. Results showed that our algorithm can successfully maintain high thermal comfort, while reducing 18% of energy consumption.