The massive deployment of small cells in 5G networks represents an alternative to meet the ever increasing mobile data traffic and to provide very-high throughout by bringing the users closer to the Base Stations (BSs). This large increase in the number of network elements demands a significant increase in the energy consumption and carbon footprint followed by complex interference management. In order to address these challenges, we consider multi-level Sleep Mode (SM) where BS components with similar activation/deactivation times can be put to sleep. The deeper and higher energy efficient the SM is, the longer it will take the BS to activate, which might impose degradation in the Quality of Service (QoS). While this adds operational flexibility to the BS, it brings complex management to the operator. In this paper, we consider a heterogeneous network architecture where small cells can switch to different SM levels to save energy and reduce dropping rate. We propose a reinforcement learning algorithm for small cells that adapts their activities subject to service delay constraint. In this regard, the algorithm intelligently learns from the environment based on the co-channel interference, the cell buffer size and the expected cell throughput in order to decide the best SM policy. Numerical values show that important energy savings can be obtained with an acceptable dropping rate. Moreover, we show that while offloading users to the macro cell can significantly reduce their delay, dropping rate and the cluster energy consumption, it comes at a cost of decreasing the network energy efficiency up to 5 times compared with the case of no offload.