Real-time detection of workers is crucial in construction safety management. Deep learning-based detecting methods are valuable, but always challenged by the possibility of target missing or identity errors under complex scenarios. To address these limitations, previous research depended on re-training for new models or datasets, which are prohibitively time-consuming and incur high computing demands. However, we demonstrate that the better detecting model might not rely on more re-training of weights; instead, a training-free model can achieve even better performance by integrating head information. In this paper, a new head-detecting branch (55 MB) is added to the Keypoint Region-based Convolutional Network (Keypoint R-CNN, 226 MB) without altering its original weights, allowing for a less occluded head to aid in body detection. We also deployed motion information and anthropometric data through a post-processing module to calculate movement relationships. This study achieved an identity F1-score (IDF1) of 97.609%, recall (Rcll) of 98.173%, precision (Prcn) of 97.052%, and accuracy of 95.329% as a state-of-the-art (SOTA) method for worker detection. This exploration breaks the inertial attitudes of re-training dependency and accelerates the application of universal models, in addition to reducing the computational difficulty for most construction sites, especially in scenarios with an insufficient graphics processing unit (GPU). More importantly, this study can address occlusion challenges effectively in the worker detection field, making it of practical significance.