In power grid construction projects, ensuring the safety of construction workers by eliminating potential risks has always been important yet difficult. Thanks to the breakthrough in deep learning, it becomes possible to adopt deep learning based object detection technologies to enable intelligent safety monitoring on power grid construction sites. However, due to the complex terrain of the power grid construction site, there is a lack of a dataset of power grid construction scenes. In this paper, a high-quality power grid construction dataset is constructed. In order to reduce the expensive annotation work, we propose to adopt contrastive self-supervised learning to pretrain a feature extraction encoder and integrate it with YOLOv5 by modifying its backbone network for safety-related object detection. The proposed network is referred to as Contrastive Res-YOLOv5 Network (CORY-Net). Experimental results show that contrastive self-supervised pretraining can efficiently improve the object detection performance on the power grid construction dataset and speed up the convergence of the model. With only 1400 pretraining epochs, the proposed CORY-Net outperforms the original YOLOv5 by contributing 1.32% to the mAP@.5 and 3.54% to the mAP@.5:.95. Compared to the supervised object detection benchmarks, the proposed CORY-Net achieves the highest mAP@.5 in 9 out of 22 categories of targets and the highest overall mAP@.5.