Novel memristive hyperchaotic system designs and their engineering applications have received considerable critical attention. In this paper, a novel multistable 5D memristive hyperchaotic system and its application are introduced. The interesting aspect of this chaotic system is that it has different types of coexisting attractors, chaos, hyperchaos, periods, and limit cycles. First, a novel 5D memristive hyperchaotic system is proposed by introducing a flux-controlled memristor with quadratic nonlinearity into an existing 4D four-wing chaotic system as a feedback term. Then, the phase portraits, Lyapunov exponential spectrum, bifurcation diagram, and spectral entropy are used to analyze the basic dynamics of the 5D memristive hyperchaotic system. For a specific set of parameters, we find an unusual metastability, which shows the transition from chaotic to periodic (period-2 and period-3) dynamics. Moreover, its circuit implementation is also proposed. By using the chaoticity of the novel hyperchaotic system, we have developed a random number generator (RNG) for practical image encryption applications. Furthermore, security analyses are carried out with the RNG and image encryption designs.