This study aimed to explore factors associated with poor quality of sleep in construction workers. This study was cross-sectional, correlational in design and used secondary data from fatigue instrument development study. We analyzed the data from 206 participants aged over 19 years who worked at construction sites for more than 6 months. We used multivariate binary logistic regression to identify the factors associated with poor quality of sleep. We classified the two sleep quality groups based on the Pittsburgh Sleep Quality Index (PSQI) score, and almost 63% of them were classified as the poor quality of sleep group. Based on multivariate binary logistic regression (Cox and Snell R2 = 0.317, Nagelkerke R2 = 0.429), the poor quality of sleep group tended to sleep for a shorter duration before the working day, and not only showed lower sleep latency and higher levels of daytime dysfunction and discomfort in daily life, but also had more chronic disease, depressive symptoms, and higher physical fatigue. Our study findings support that there are many modifiable factors associated with poor sleep and a high rate of poor quality of sleep occurred in construction workers. Thus, clinicians should consider providing diverse options for applying interventions to ensure better sleep, fatigue management, and depression prevention in construction workers after considering their unique characteristics.