The timing of food intake can significantly alter the body’s metabolism of nutrient intake and affect the occurrence of chronic diseases. However, whether and how the intake time of dietary fiber could influence mortality risks is largely unknown. This study aims to reveal the association between total dietary fiber intake and fiber intake at different times with all-cause, cancer, and cardiovascular disease (CVD) mortality rates. A total of 31,164 adults who enrolled in the National Health and Nutrition Examination Survey from 2003 to 2014 are included in this study. Dietary fiber intake was measured using 2-day, 24 h dietary recall. The main exposures in this study were the intake of dietary fiber at breakfast, lunch, and dinner via regression analysis of the residual method. The main outcomes were the all-cause, cancer, and CVD mortality rates. Cox proportional hazards regression models were used to evaluate the survival relationship between dietary fiber intake at different times and mortality rates. Among the 31,164 adults, 2915 deaths, including 631 deaths due to cancer and 836 deaths due to CVD, were documented. Firstly, after adjusting for potential confounders, compared to the participants in the lowest quintile of total dietary fiber intake, the participants in the highest quintile of fiber intake had lower all-cause (HR = 0.686, 95% CI: 0.589–0.799, p for trend <0.001) and cancer (HR = 0.606, 95% CI: 0.446–0.824, p for trend = 0.015) mortality risks. Secondly, compared to the participants in the lowest quintile of dietary fiber intake at dinner, the participants in the highest quintile of fiber intake had lower all-cause (HR = 0.796, 95% CI: 0.668–0.949, p for trend = 0.009) and cancer (HR = 0.564, 95% CI: 0.388–0.822, p for trend = 0.005) mortality risks. Furthermore, equivalently replacing each standard deviation of dietary fiber consumed at breakfast with that at dinner was associated with lower cancer mortality risks (HR = 0.846, 95% CI: 0.747–0.958). In conclusion, this study demonstrates that, in the NHANES (2003–2014) cohort, to reduce all-cause and cancer mortality risks, the optimal dietary fiber intake time is in the evening.