Access networks require higher bandwidth and throughput to handle rapidly increasing traffic. However, in the time-division multiplexing passive optical network (TDM-PON) system, improving system throughput in such a time-slotted network faces two technical challenges. First, because optical network units transmit packets with individual clock frequency, receivers at the optical line terminal side must instantly recover the frequency to guarantee the throughput, while no valid data can be received before the clock and data recovery (CDR) is completed. Second, the time-slotted mechanism requires fine-granularity of time-slot edge alignment, where the interpacket interval used to compensate the misalignment deteriorates the throughput. In this paper, these two challenges are simultaneously addressed by white rabbit (WR) protocol-based enhancements for the first time, to our best knowledge, in a TDM-PON system. With the synchronous Ethernet of WR protocol, the physical-layer frequency synchronization efficiently accelerates the CDR via eliminating the time-consuming frequency tracking process. Moreover, after improving the time precision to the subnanosecond level via a digital dual mixer time difference phase detector, the precise delay model in the WR protocol can accurately elaborate the compensation and adjustment on time. The experimental results validate that the proposed network achieves 38.6 ns CDR time and 341 ps time synchronization precision.