Vital signs monitoring in physical activity (PA) is of great significance in daily healthcare. Impulse Radio Ultra-WideBand (IR-UWB) radar provides a contactless vital signs detection approach with advantages in range resolution and penetration. Several researches have verified the feasibility of IR-UWB radar monitoring when the target keeps still. However, various body movements are induced by PA, which lead to severe signal distortion and interfere vital signs extraction. To address this challenge, a novel joint chest–abdomen cardiopulmonary signal estimation approach is proposed to detect breath and heartbeat simultaneously using IR-UWB radars. The movements of target chest and abdomen are detected by two IR-UWB radars, respectively. Considering the signal overlapping of vital signs and body motion artifacts, Empirical Wavelet Transform (EWT) is applied on received radar signals to remove clutter and mitigate movement interference. Moreover, improved EWT with frequency segmentation refinement is applied on each radar to decompose vital signals of target chest and abdomen to vital sign-related sub-signals, respectively. After that, based on the thoracoabdominal movement correlation, cross-correlation functions are calculated among chest and abdomen sub-signals to estimate breath and heartbeat. The experiments are conducted under three kinds of PA situations and two general body movements, the results of which indicate the effectiveness and superiority of the proposed approach.