It has been shown that the heating, ventilation, and air conditioning (HVAC) systems of commercial buildings can offer ancillary services to power systems without loss of comfort. In this paper, we propose a new control framework for reliable scheduling and provision of frequency reserves by aggregations of commercial buildings. The framework incorporates energy-constrained frequency signals, which are adopted by several transmission system operators for loads and storage devices. We use a hierarchical approach with three levels: (i) reserve capacities are allocated among buildings (e.g., on a daily basis) using techniques from robust optimization, (ii) a robust model predictive controller optimizes the HVAC system consumption typically every 30 minutes, and (iii) a feedback controller adjusts the consumption to provide reserves in real time. We demonstrate how the framework can be used to estimate the reserve capacities in simulations with typical Swiss office buildings and different reserve product characteristics. Our results show that an aggregation of approximately 100 buildings suffices to meet the 5 MW minimum bid size of the Swiss reserve market.