Insecure networks are vulnerable to cyber-attacks, which may result in catastrophic damages on the local and global scope. Nevertheless, one of the tedious tasks in detecting any type of attack in a network, including DoS attacks, is to determine the thresholds required to discover whether an attack is occurring or not. In this paper, a hybrid system that incorporates different heuristic techniques along with a Finite State Machine is proposed to detect and classify DoS attacks. In the proposed system, a Genetic Programming technique combined with a Genetic Algorithm are designed and implemented to represent the system core that evolves an optimized tree-based detection model. A Hill-Climbing technique is also employed to enhance the system by providing a reference point value for evaluating the optimized model and gaining better performance. Several experiments with different configurations are conducted to test the system performance using a synthetic dataset that mimics real-world network traffic with different features and scenarios. The developed system is compared to many state-of-art techniques with respect to several performance metrics. Additionally, a Mann-Whitney Wilcoxon test is conducted to validate the accuracy of the proposed system. The results show that the developed system succeeds in achieving higher overall performance and prove to be statistically significant.