The secure transmission of information is one of the most important topics in the field of information technology. Considering that images contain important visual information, it is crucial to create a safe platform for image transfer. One commonly employed tool to enhance the complexity and randomness in image encryption methods is the chaos system. The logistic and sine maps are utilized in encryption algorithms but these systems have some weaknesses, notably chaotic behavior in a confined area. In this study, to address these weaknesses, a hybrid system based on the Atangana–Baleanu fractional derivative is proposed. The various tests employed to evaluate the behavior of the new system, including the NIST test, histogram analysis, Lyapunov exponent calculation, and bifurcation diagram, demonstrate the efficiency of the proposed system. Furthermore, in comparison to the logistic and sine maps, the proposed hybrid exhibits chaotic behavior over a broader range. This system is utilized to establish a secure environment for the transmission of multiple images within an encryption algorithm, subsequently concealing them within a meaningful image. Various tools employed to assess the security of the proposed algorithm, including histogram analysis, NPCR, UACI, and correlation values, indicate that the proposed hybrid system has application value in encryption.