Abstract. The observation of the nocturnal boundary layer height (NBLH) plays an important role in air pollution and monitoring. Through 39 d of heavy pollution observation experiments in Beijing (China), as well as an exhaustive evaluation of the gradient, wavelet covariance transform, and cubic root gradient methods, a novel algorithm based on the cluster analysis of the gradient method (CA-GM) of lidar signals is developed to capture the multilayer structure and achieve night-time stability. The CA-GM highlights its performance compared with radiosonde data, and the best correlation (0.85), weakest root-mean-square error (203 m), and an improved 25 % correlation coefficient are achieved via the GM. Compared with the 39 d experiments using other algorithms, reasonable parameter selection can help in distinguishing between layers with different properties, such as the cloud layer, elevated aerosol layers, and random noise. Consequently, the CA-GM can automatically address the uncertainty with multiple structures and obtain a stable NBLH with a high temporal resolution, which is expected to contribute to air pollution monitoring and climatology, as well as model verification.