Objective: The aim of this study was to confirm the association between working hours and self-rated health, and to find the degree of changes in health level by working hours according to gender. Methods: This study was based on the 929 workers (571 men and 358 women) from the Korean Labor and Income Panel Study during 2004–2006. To minimize the healthy worker effects, the study subjects included only those who did not have any chronic diseases, and who answered their health status as “moderate” or above in the baseline. Logistic regression analysis was used to confirm the associations between working hours and self-rated health. Results: In men, working hours per week of 47–52 h, 53–68 h, and >68 h were associated with 1.2, 1.3, and 1.1 times increases, respectively, in the odds ratio on worsened self-rated health, compared with the reference group (40–46 h). On the other hand, the risks were 1.0, 2.2, and 2.6 times increases in women. However, the results were different according to gender in the group with less than 40 h. The men with less than 40 h had a 0.9 times odds ratio on worsened self-rated health. For the women with less than 40 h, the odds ratio on self-rated health was 5.4 times higher than the reference group. Conclusions: Working more than 52 h per week had a negative effect on health, regardless of gender. However, in the group with less than 40 h, the negative association between working hours and self-rated health were shown only in women. Health outcomes due to working hours may differ by gender. Therefore, further studies are needed to explore the causes of these results.