Next generation satellite communications (NGSC) evolved by 5G NR combination is the hot issue in recent researches. For the conditions of new frequency (Ku/Ka/Q/V) and much more broadband (>100MHz) for satellite mobile communications, the obvious nonlinearity by Input Multiplexing (IMUX) filter, High Power Amplifier (HPA), and Output Multiplexing (OMUX) filter will cause serious signals degradation. Traditional behavioral modeling methods of IMUX filter, HPA, and OMUX filter (IHO) are relatively independent of each other, which is difficult to quantitatively evaluate intrinsic modeling errors and cumulative modeling errors. That makes modeling errors of IHO cannot be compensated accurately. In this paper, we are the first to exploit a quantitative evaluation of cascade modeling errors for IHO. A novel accurate and lowcomplexity behavior model for IHO of LEO satellite based on the iterative compensation of cascade modeling errors (ICCE-IHO) is proposed. Based on the normalized least mean square (NLMS) algorithm, the ICCE-IHO model realizes accurate modeling and low-complexity. Experimental results show that the ICCE-IHO model can achieve a maximum 0.67dB improvement in Normalized Mean Square Error (NMSE), a maximum 2.24dB improvement in Adjacent Channel Error Power Ratio (ACEPR), and the hardware resource consumption is reduced by 28%.