The ongoing evolution of optical networks towards autonomous systems supporting high-performance services beyond 5G requires advanced functionalities for automated security management. These functionalities need to support risk reduction, security diagnostics and incident remediation strategies. To cope with evolving security threat scenarios, security diagnostic approaches should be able to detect and identify the nature not only of existing attack techniques, but also those hitherto unknown or insufficiently represented. Machine Learning (ML)-based algorithms have been shown to perform well when identifying known attack types, but cannot guarantee precise identification of unknown attacks. This makes Root Cause Analysis (RCA) a crucial tool to enable timely attack response when human intervention is unavoidable.We address these challenges by establishing an ML-based framework for security assessment and analyzing RCA alternatives for physical-layer attacks. We first scrutinize different Network Management System (NMS) architectures and the corresponding ML-based security assessment functionalities. We then investigate the applicability of supervised and unsupervised learning (SL and UL) approaches for RCA and propose a novel UL-based RCA algorithm called Distance-Based Root Cause Analysis (DB-RCA). Extensive validation of the framework's applicability and performance in the context of autonomous optical network security management is carried out using an experimental physical-layer security dataset, evaluating the benefits and drawbacks of the SL-and UL-based RCA techniques. Besides confirming that SL-based approaches can be trained to provide precise RCA output for known attack types, the study shows that the proposed UL-based RCA approach offers meaningful insights into the properties of anomalies caused by novel attack types, thus supporting the human security officers in advancing the physical-layer security diagnostics.