SummaryCloud computing is distributed type of technology to offers distant services by the internet to manage, access, and store data on their server. The cloud server manages and distributes large amounts of data. During large amounts of complex structured and unstructured data transactions, in order to protect the data and cloud resources from harmful activities by using the main task of the security system. Therefore, a novel intrusion detection system is required in cloud networks to detect malicious activity and improve secure data transmission. In this paper, a novel technique called advanced set containment deep learned Rabin certificateless signcryption (ADRES) technique is employed to secure data transmission for achieving better confidentiality, and integrity, in cloud computing. The ADRES technique includes three major phases namely Registration and key generation, signcryption, and unsigncryption. At the first phase, the user registers details to the cloud server using secure transmission in big data analytics. The cloud server generates the private and public keys for each registered user. In the second phase, the signcryption process is determined by cloud user that encrypts and generates the signature by using the Rabin cryptosystem. Then encrypted data is stored in a cloud server with minimum space complexity. Finally, the unsigncryption is performed by using signature verification and decryption. First, the signature verification is performed by applying an Advanced set containment Deep belief neural network for identifying the authorized or unauthorized user (i.e., attacker). The authorized user decrypts data using the Rabin cryptosystem to achieve the original data. Thus, secure data transmission is obtained through improved data confidentiality and integrity. Simulation of the proposed ADRES technique is performed with respect to data confidentiality level, data integrity rate, and computation time, space complexity with several numbers of data. Experimental results reported ADRES obtains the improved data transmission security with better confidentiality rate by 7%, integrity by 7%, and lesser time 16% and space complexity by 18% than the conventional methods.