Summary
Currently, core networking architectures are facing disruptive developments, due to emergence of paradigms such as Software‐Defined‐Networking (SDN) for control, Network Function Virtualization (NFV) for services, and so on. These are the key enabling technologies for future applications in 5G and locality‐based Internet of things (IoT)/wireless sensor network services. The proliferation of IoT devices at the Edge networks is driving the growth of all‐connected world of Internet traffic. In the Cloud‐to‐Things continuum, processing of information and data at the Edge mandates development of security best practices to arise within a fog computing environment. Service providers are transforming their business using NFV‐based services and SDN‐enabled networks. The SDN paradigm offers an easily programmable model, global view, and control for modern networks, which demand faster response to security incidents and dynamically enforce countermeasures to intrusions and cyberattacks. This article proposes an autonomic multilayer security framework called Distributed Threat Analytics and Response System (DTARS) for a converged architecture of Fog/Edge computing and SDN infrastructures, for emerging applications in IoT and 5G networks. The major detection scheme is deployed within the data plane, consisting of a coarse‐grained behavioral, anti‐spoofing, flow monitoring and fine‐grained traffic multi‐feature entropy‐based algorithms. We developed exemplary defense applications under DTARS framework, on a malware testbed imitating the real‐life DDoS/botnets such as Mirai. The experiments and analysis show that DTARS is capable of detecting attacks in real‐time with accuracy more than 95% under attack intensities up to 50 000 packets/s. The benign traffic forwarding rate remains unaffected with DTARS, while it drops down to 65% with traditional NIDS for advanced DDoS attacks. Further, DTARS achieves this performance without incurring additional latency due to data plane overhead.