Passive optical network (PON) is one of the key enabling technologies to fulfill the latest exceptional bandwidth demand owing to an exponential rise of Internet data traffic induced by the expansion of bandwidth‐hungry applications services. The potential of using passive star topology for ITU‐T G.989.x standardardized next generation PON incorporating vertical cavity surface emitting laser input source, is proposed in this paper. Mode division multiplexing (MDM) technique employing Hermite‐Gaussian (HG10 and HG20) modes are used to enhance the channel capacity at bidirectional 80 Gbps traffic rate. By utilizing polarization mode dispersion (PMD) emulator, this proposed system is promising since it offers good tolerance ability of anti‐PMD, anti‐dispersion, suppression of nonlinear effects and high spectrum effectiveness at high‐speed transmission. Impressively, faithful 600 and 590 m range is obtained at HG10 and HG20 modes respectively, with clear eye patterns at bit error rate of 10−9 for 1596–1598.4 nm in downlink. In uplink, faithful 600 and 530–600 m distance is achieved for 1524–1526.4 nm at HG10 and HG20 respectively. For variable laser temperature, maximum tolerable temperature of 20°C for both modes in downlink and 20°C at HG10 as well as 14C at HG20 is observed in uplink, over 600 m range. Besides, the proposed design undergoes ∼1e‐30 at HG10 and ∼1e‐35 at HG20 coupling coefficients for linearly polarized (LP[0,1] to LP[−4,3]) modes. It is also analyzed that maximum 3.99 dB gain and 84.58 dB optical signal‐to‐noise ratio is obtained for the proposed scheme. This design does not suffer from shot, thermal and phase noise and offers optimum performance than existing systems.