The active coherent polarization beam combining (CPBC) technique has been experimentally proved to be a promising approach for the energy and power scaling of ultrashort laser pulses, despite the tremendous challenge in temporal synchronization, dispersion management and nonlinearity control. In order to develop a comprehensive theoretical model to investigate the influence of temporal-spectral effects on ultrafast fiber active CPBC systems, a generalized nonlinear Schrödinger equation carrying spectral factors is used to depict the propagation of ultrashort pulses in fiber amplifier channels and ultrashort-pulsed Gaussian beams (PGBs) carrying temporal-spatial factors are utilized to picture the propagation of ultrashort pulses in the free space. To the best of our knowledge, the influence of different temporal-spectral effects has been segregated for the first time and corresponding analytical equations have been strictly derived to link the combining efficiency with specific factors. Based on our analysis, the optical path difference (OPD) has the most detrimental impact on the combining efficiency because of the high controlling accuracy and anti-interference requirements. For instance, the OPD must be controlled in ~ ±14 μm to achieve a combining efficiency of above 95% for combining ultrashort laser pulses with a 3 dB spectral bandwidth of 13 nm centered at 1064 nm. Besides, the analytical expression also demonstrates that the impact of self-phase modulation on the combining efficiency has no dependence on spectral bandwidth and only depends on the B integral difference if neglecting the direct influence of the peak power difference. Our analysis also indicates that the group velocity dispersion has relatively small influence on the combining efficiency. These formulas can be used to diagnose the influence of temporal-spectral effects and provide useful guidelines for the design or optimization of the active CPBC system of ultrafast fiber chirped-pulse amplifiers.