Aiming at the jitter issue in four-level pulse amplitude modulation (PAM4) receive link, the key causes related to the non-ideal effects of channel and data decision were analyzed, and a modified PAM4 receiver architecture was proposed. An analog equalizer with extra mid-frequency compensation and 1-tap decision-feedback equalizer were incorporated to minimize the inter-symbol interference. To alleviate the timing constraints in DFE, the feedback loop of slicer was optimized. The post-simulation results based on TSMC 28 nm CMOS process indicate that the proposed 50 Gb/s PAM4 receiver functions well over a channel with 12 dB loss at Nyquist frequency. The peak-to-peak jitters of two restored NRZ signals are 1.5 ps and 2.5 ps, respectively.