This paper proposes a novel tweak-cube color image encryption scheme jointly manipulated by chaos and hyper-chaos. One-dimensional (1D) chaotic maps are effortless to operate, but the key space is relatively small. The hyperchaotic system has complex dynamics properties, which are capable of compensating for the defects of 1D chaotic maps. Thus, we first raise an improved 1D chaotic map with an increased key space. Then, we associate it with a four-dimensional (4D) hyperchaotic system to generate the key streams and further rotate and shift the rows and columns of each component of Red (R), Green (G), and Blue (B) for the color image. The permuting mode is to disturb the original position of the pixels by mimicking the way of twisting the Rubik’s cube. Moreover, the key stream updated by the plain images is also utilized for diffusion and scramble at the bit level. As a consequence, our cryptosystem enhances the security without at the expense of increasing time cost.