In this paper, a novel full duplex Radio Frequency Passive Optical Network (RPON) system by using 8-ary pulse amplitude modulation (8-PAM) downlink and duobinary (DB) uplink signals is designed. We measure optical spectrum diagrams, eye diagrams, bit error rate (BER), and analyze the reception performance of the 10Gbit/s system before and after transmission. Results show that the downlink 8-PAM and uplink DB signals can improve bandwidth efficiency. Hence, the scheme is expected to be applied in the future high-speed full duplex access system.