Taking vapor-liquid separators used in marine desalination as objects, a novel kind of plate-type vapor-liquid separator with groove is designed to achieve a more prominent separation effect. The Eulerian-Lagrangian approach is applied to simulate the two-phase flow in the separator. The SST j-x turbulence model is utilized to simulate the continuous phase, and discrete phase model (DPM) is employed to calculate the water droplet distribution and the corresponding trajectories, which is verified by the experimental results from the literature. The impacts from the groove height, width and depth on the separator performance are studied. Furthermore, to acquire the optimal geometry of the separators with groove, a dimensionless objective function considering the space size, separation efficiency and pressure drop is proposed, and the corresponding prediction model is established based on the response surface methodology. Compared to the traditional type, the computational fluid dynamic (CFD) simulation results show that a higher velocity and bigger circulation region contribute to a much more comprehensive performance in the proposed separator. After determining the optimal values for all the parameters, a discrepancy value of 5.2% between the simulation results and the prediction value is attained, which indicates that the prediction model is applicable to guide the design of the separator. Further study shows that the optimal novel separator guarantees the separation efficiency with much smaller size compared with the original one.