Construction safety requires real-time monitoring due to its hazardous nature. Existing vision-based monitoring systems classify each frame to identify safe or unsafe scenes, often triggering false alarms due to object misdetection or false detection, which reduces the overall monitoring system’s performance. To overcome this problem, this research introduces a safety monitoring system that leverages a novel temporal-analysis-based algorithm to reduce false alarms. The proposed system comprises three main modules: object detection, rule compliance, and temporal analysis. The system employs a coordination correlation technique to verify personal protective equipment (PPE), even with partially visible workers, overcoming a common monitoring challenge on job sites. The temporal-analysis module is the key component that evaluates multiple frames within a time window, triggering alarms when the hazard threshold is exceeded, thus reducing false alarms. The experimental results demonstrate 95% accuracy and an F1-score in scene classification, with a notable 2.03% average decrease in false alarms during real-time monitoring across five test videos. This study advances knowledge in safety monitoring by introducing and validating a temporal-analysis-based algorithm. This approach not only improves the reliability of safety-rule-compliance checks but also addresses challenges of misdetection and false alarms, thereby enhancing safety management protocols in hazardous environments.