In this paper, we introduce a novel approach for detecting Denial of Service (DoS) attacks in software-defined IP over optical networks, leveraging machine learning to analyze optical spectrum features. This method employs machine learning to automatically process optical spectrum data, which is indicative of network security status, thereby identifying potential DoS attacks. To validate its effectiveness, we conducted both numerical simulations and experimental trials to collect relevant optical spectrum datasets. We then assessed the performance of three machine learning algorithms XGBoost, LightGBM, and the BP neural network in detecting DoS attacks. Our findings show that all three algorithms demonstrate a detection accuracy exceeding 97%, with the BP neural network achieving the highest accuracy rates of 99.55% and 99.74% in simulations and experiments, respectively. This research not only offers a new avenue for DoS attack detection but also enhances early detection capabilities in the underlying optical network through optical spectrum data analysis.