For the train timetabling problem (TTP) in a metro system, the operator-oriented and passenger-oriented objectives are both important but partly conflicting. This paper aims to minimize both objectives by considering frequency (in the line planning stage) and train cost (in the vehicle scheduling stage). Time-varying passenger demand and train capacity are considered in a nonsmooth, nonconvex programming model, which is transformed into a mixed integer programming model with a discrete time-space graph (DTSG). A novel dwell time determining process considering congestion at stations is proposed, which turns the dwell times into dependent variables. In the solution approach, we decompose the TTP into a subproblem for optimizing segment travel times (OST) and a subproblem for optimizing departure headways from the shunting yard (OH). Branch-and-bound and frequency determining algorithms are designed to solve OST. A novel rolling optimization algorithm is designed to solve OH. The numerical experiments include case studies on a short metro line and Beijing Metro Line 4, as well as sensitivity analyses. The results demonstrate the predictive ability of the model, verify the effectiveness and efficiency of the proposed approach, and provide practical insights for different scenarios, which can be used for decision-making support in daily operations.