The present investigation devices a novel X-type six-high (X-6h) mill. In addition, parametric models of different roll layouts such as the four-high (4-h), I-type six-high (I-6h), and X-6h mills are established. Three-dimensional (3D) finite element (FE) contact analysis for a strip rolling process is conducted when the mills are subjected to a constant vertical load of 65 kN. Through comparative analysis of von Mises stress, contact stress and elastic deformation displacement in three roll layouts, the rigidity characteristic of each is obtained, and it is found that the proposed X-6h mill has the largest roll gap stiffness. The influence of different roll diameter ratios on the roll gap stiffness of the roll system is investigated, based on which an optimization design model is built. Further, by taking into account the roll gap stiffness of the roll system as the optimization objective, the optimum diameter ratios of backup roll (BUR) to work roll (WR) of the X-6h rolling mill is achieved via the genetic algorithm (GA) optimization method, obtaining the optimum structural parameters of BUR and WR as well. The reliability of the proposed design is verified by manufacturing a prototype mill which produced magnesium alloy and aluminum alloy strips of high quality.