The effect of mixing on coupled heat release and transfer performance of a novel segregated solid motor is numerically evaluated with a transient two-dimensional combustion model. The results show that vortex structures are formed and evolved in the combustion chamber. Quantitative calculation of the mixing effect shows the inhomogeneous distribution of oxidant and fuel species. The well-mixing area is located in a narrow belt-like coupled combustion region near the burning surface of the propellant. Heat transfer coefficient decreases greatly due to lower combustion reaction rate and enlarged flow channel area. Heat transfer coefficients near the two ends of the propellant grain are higher than other parts due to the influence of vortex mixing. Raising the inlet mass flow rate leads to enhanced mixing and heat transfer, which results in a lower temperature and regression rate of the propellant with combustion time. Temperature and oxidation rates of H2 and CO are unevenly distributed in the boundary layer of coupled combustion. Increasing the mass flux of inlet oxidizer gas leads to a higher combustion heat release rate. Therefore, the gas-phase temperature increases significantly. The heat release rate reaches the maximum near the ends of the propellant grain, where vortex mixing strengthens the coupled combustion process in the motor.