The expanding and ubiquitous availability of the Internet of Things (IoT) have changed everyone's life easier and more convenient. Same time it also offers a number of issues, such as effectiveness, security, and excessive power consumption, which constitute a danger to intelligent IoT-based apps. Group managing is primarily used for transmitting and multi-pathing communications that are secured with a general group key and it can only be decrypted by an authorized group member. A centralized trustworthy system, which is in charge of key distribution and upgrades, is used to maintain group keys. To provide longitudinal access controls, Software Defined Network (SDN) based security controllers are employed for group administration services. Cloud service providers provide a variety of security features. There are just a few software security answers available. In the proposed system, a hybrid protocols were used in SDN and it embeds edge system to improve the security in the group communication. Tree-based algorithms compared with Group Key Establishment (GKE) and Multivariate public key cryptosystem with Broadcast Encryption in the proposed system. When all factors are considered, Broadcast Encryption (BE) appears to become the most logical solution to the issue. BE enables an initiator to send encrypted messages to a large set of recipients in a efficient and productive way, meanwhile assuring that the data can only be decrypted by defining characteristic. The proposed method improves the security, efficiency of the system and reduces the power consumption and minimizes the cost.