The network system has become an indispensable component of modern infrastructure. DDoS attacks and their variants remain a potential and persistent cybersecurity threat. DDoS attacks block services to legitimate users by incorporating large amounts of malicious traffic in a short period or depleting system resources through methods specific to each client, causing the victim to lose reputation, finances, and potential customers. With the advancement and maturation of artificial intelligence technology, machine learning and deep learning are widely used to detect DDoS attacks with significant success. However, traditional supervised machine learning must depend on the categorized training sets, so the recognition rate plummets when the model encounters patterns outside the dataset. In addition, DDoS attack techniques continue to evolve, rendering training based on conventional data models unable to meet contemporary requirements. Since closed-set classifiers have excellent performance in cybersecurity and are quite mature, this study will investigate the identification of open-set recognition issues where the attack pattern does not accommodate the distribution learned by the model. This research proposes a framework that uses reconstruction error and distributes hidden layer characteristics to detect unknown DDoS attacks. This study will employ deep hierarchical reconstruction nets (DHRNet) architecture and reimplement it with a 1D integrated neural network employing loss function combined with spatial location constraint prototype loss (SLCPL) as a solution for open-set risks. At the output, a one-class SVM (one-class support vector machine) based on a random gradient descent approximation is used to recognize the unknown patterns in the subsequent stage. The model achieves an impressive detection rate of more than 99% in testing. Furthermore, the incremental learning module utilizing unknown traffic labeled by telecom technicians during tracking has enhanced the model’s performance by 99.8% against unknown threats based on the CICIDS2017 Friday open dataset.