The effect of nonsinusoidal oscillation at different modification ratios (α) on slag lubrication was investigated during mold oscillation. A validated and reliable multiphase model was employed, which involved flow and solidification of the molten steel and mold slag. The main results revealed that a large amount of liquid slag at the entrance of the mold–strand channel reflowed into the slag pool at the middle of the negative strip period. The phenomenon was more distinct, with an increase in the modification ratio. The modification ratio had no obvious effect on the average thickness of the liquid film at different depths below the meniscus. A modification ratio of 0.5 caused less fluctuation of the transient liquid film. Quantitative prediction of slag consumption indicated that as the modification ratio increased from 0.2 to 0.5 to 0.8, the average values were 0.278, 0.286, and 0.279 kg/m2, respectively. Shell solidification and growth near the meniscus mainly occurred when the mold was descending, which not only depended on the heat flux, but also on the liquid slag flow, the pressure driven by slag rim, and the mold oscillation. Optimization of the modification ratio of nonsinusoidal oscillation could be an alternative to delay growth of the initial shell towards the molten steel. A modification ratio of 0.5 had the least robust shell tip at the meniscus, thereby reducing entrapment of inclusions and bubbles by the shell tip.