Cardiorespiratory (e.g., heart rate and breathing rate) and thermoregulatory (e.g., local skin temperature and electrodermal activity) responses are controlled by the sympathetic nervous system. To cope with increased physical workload, the sympathetic system upregulates its activity to generate greater sympathetic responses (i.e., increased heart rate and respiratory rate). Therefore, physiological measures may have the potential to evaluate changes in physical condition (including fatigue) during functional tasks. This study aimed to quantify physical fatigue using wearable cardiorespiratory and thermoregulatory sensors during a simulated construction task. Twenty-five healthy individuals (mean age, 31.8 ± 1.8 years) were recruited. Participants were instructed to perform 30 min of a simulated manual material handling task in a laboratory. The experimental setup comprised a station A, a 10-metre walking platform, and a station B. Each participant was asked to pick up a 15 kg ergonomically-designed wooden box from station A and then carried it along the platform and dropped it at station B. The task was repeated from B to A and then A to B until the participants perceived a fatigue level > 15 out of 20 on the Borg-20 scale. Heart rate, breathing rate, local skin temperature, and electrodermal activity at the wrist were measured by wearable sensors and the perceived physical fatigue was assessed using the Borg-20 scale at baseline, 15 min, and 30 min from the baseline. There were significant increases in the heart rate (mean changes: 50 ± 13.3 beats/min), breathing rate (mean changes: 9.8 ± 4.1 breaths), local skin temperature (mean changes: 3.4 ± 1.9 °C), electrodermal activity at the right wrist (mean changes: 7.1 ± 3.8 µS/cm), and subjective physical fatigue (mean changes: 8.8 ± 0.6 levels) at the end of the simulated construction task (p < 0.05). Heart rate and breathing rate at 15 and 30 min were significantly correlated with the corresponding subjective Borg scores (p < 0.01). Local skin temperature at 30 min was significantly correlated with the corresponding Borg scores (p < 0.05). However, electrodermal activity at the right wrist was not associated with Borg scores at any time points. The results implied cardiorespiratory parameters and local skin temperature were good surrogates for measuring physical fatigue. Conversely, electrodermal activity at the right wrist was unrelated to physical fatigue. Future field studies should investigate the sensitivity of various cardiorespiratory and thermoregulatory parameters for real time physical fatigue monitoring in construction sites.