Optical burst switching (OBS) is a network architecture that combines the advantages of packet and circuit switching techniques. However, OBS networks are susceptible to cyber-attacks, such as flooding attacks, which can degrade their performance and security. This paper introduces a novel machine learning method for flooding attack detection in OBS networks, based on a third-order distance function for k-nearest neighbors (KNN3O). The proposed distance is expected to improve detection accuracy due to higher sensitivity with respect to the distance difference between two points. The developed method is compared with seven other machine learning methods, namely standard KNN, KNN with cosine distance (KNNC), multi-layer perceptron (MLP), naive Bayes classifier (NBC), support vector machine (SVM), decision tree (DT), and discriminant analysis classifier (DAC). The methods are further assessed using five metrics: accuracy, precision, recall, F1-score, and specificity. The proposed method achieved an accuracy of 99.3%, outperforming the original KNN, MLP, and SVM, which achieved accuracies of 99%, 76.4%, and 94.7%, respectively. The results show that KNN3O is the best method for flooding attack detection in OBS networks, as it achieves the highest scores in all five metrics.