SummaryIn this paper, we propose a reactive defragmentation with minimum spectrum route (RDMSR) for the problem of route, spectrum, and modulation‐format allocation (RSMA) in mixed grid optical network. In mixed grid network, spectrum redundancy and its assignment problem increase due to the spectrum continuity and contiguity constrains. In the proposed RDMSR strategy, process of defragmentation initiates after the termination of the existing connections. In addition, the route that needs minimum spectrum is given priority over the other available routes. The performance of the proposed strategy is compared with the two existing strategies: shortest path first (SPF) and mixed grid aware dynamic resources allocation (MDRA). Simulations are performed on NSFNET and ARPANET topologies. The existing and proposed strategies are evaluated on the metrics of bandwidth blocking probability (BBP), network capacity (NC), and average hops (AH) at three different combinations of the network traffic. Results show the proposed strategy outperforms than the other existing state of art strategies.