Network operators heavily depend on security services to secure their information technology infrastructures. On the other hand, due to the complexity of security policies, it is not appropriate to straightforwardly use previous pathwise enforcement approaches. In this paper, the enforcement problem of the security policy on middleboxes is formulated as a weighted K set covering problem that requires a policy space analysis tool. This tool is intended to be supported on range-represented hyperrectangles, which are tagged using a prioritized R-tree. This methodological work initially evaluates the topological features of diverse types of policies. Hybrid firefly bat algorithm-supported heuristic information shows the inherent difficulties of security policies and provides direction for the design of the enforcement algorithm. At the same time, a scopewise policy enforcement procedure is proposed, which requires a moderate number of enforcement network nodes for organizing the various policy subsets in a greedy manner. Our results demonstrate that the proposed hybrid firefly bat algorithm with policy space analysis offer greatly improved outcomes in terms of the rule overhead, network security, packet delivery ratio, packet loss ratio, and time efficiency above the set operations of the security policy. KEYWORDScomputer network, fuzzy rule, hybrid firefly bat algorithm, policy space analysis, security | INTRODUCTIONThe work performed by a network involves concepts such as security examination to protect their information technology environment. Diverse types of network security strategies are found worldwide and are dispersed among numerous security middleboxes organized in networks. Nevertheless, accomplishing high performance and implementing the security advantages of middleboxes are extremely difficult tasks. This difficulty thus demands a careful sketch of the network topology, which physically implements rules to direct the traffic in the preferred series of middleboxes and performs the exact procedure when malfunctions and overloads occur. 1 Software-defined networking (SDN) presents a promising choice for policy enforcement through sensibly centralized management, decoupling of information and control planes, and the capability to programmatically organize the forwarding rules. 2

Summary Network operators depend on security services with the aim of safeguarding their IT infrastructure. Various types of network security policies are employed on a global scale and are disseminated among several security middleboxes implemented in networks. But, owing to the complications in security policies, it is not quite efficient to directly use the path‐wise enforcement schemes that are prevalent. The major motivation of this work is to improve security levels and solve the policy enforcement problem. For the first time, this work reports the issue of policy enforcement on middleboxes. The major contribution of this work is to design security policy enforcement as a Weighted K Set Covering Problem, and we designed a Policy Space Analysis (PSA) tool intended for a group of operations in the security policy. This PSA tool was developed based on range‐signified hyper‐rectangles, which are indexed by the Hilbert R‐tree. Leveraging the PSA, we first investigated the topological features of various kinds of policies. Balancing the PSA tool in a non‐dominated sorting particle swarm optimization technique exposes the intrinsic difficulties of this security strategy and provides guidance for designing the enforcement approach. In addition, in this research, a new fuzzy rule‐based classification system is introduced for packet classification. A scope‐wise policy enforcement algorithm was proposed, which chooses a moderate number of enforcement network nodes for deploying multiple policy subsets in a greedy manner. This scheme is much quicker compared with the first one and therefore has found its application in real‐time deployments.