Modern cryptography is a key element of data security, ensuring the confidentiality and integrity of information. In an increasingly digital world, cryptography remains crucial for the protection of sensitive data. In this context, we propose a novel hybrid security system for encrypted color images using a DNA model, chaotic systems, and SHA256-MD5 hash functions as a basis. The proposed hybrid system includes DNA permutation and diffusion. In DNA permutation, we unpredictably rearrange the location of DNA image elements by using the logistic map of low computational complexity. In DNA diffusion, we diffuse the permuted image of DNA with the DNA image key generated by a 5D hyper-chaotic system, using a variety of algebraic operators such as the circular offset in both directions. Considering the experimental outcomes and security evaluation, we can infer that the proposed hybrid security system demonstrates a high level of security, resistance to existing attacks, and practical application suitability while maintaining speed.