As data traffic in optical fiber network increases, there has been a lot of research that explores how to increase transmission capacity. Recent works suggest the multi-dimensional optical transmission as a novel solution. Multi-dimensional optical transmission is a signal transmission technique using orthogonal dimension sources such as intensity and phase of optical signal. However, previously proposed scheme has nonlinearity issue due to high frequency components in the modulation sequence and interference issue due to the transceiver structure. These issues limit the performance enhancement of multi-dimensional optical transmission. To overcome this limitation, geometric and probabilistic constellation shaping technique is aided to optical intensity-phase-polarization multidimensional modulation in direct detection system. We used quadrature amplitude modulation -differential phase shift keying -polarized intensity rotational frequency shift keying (QAM-DPSK-PIRFSK) modulation scheme to ensure orthogonality among optical intensity, phase and polarization. However, inter-dimensional interference (IDI) still occurs during direct phase detection using Mach-Zehnder delay interferometer (MZDI). To mitigate the IDI effect, RF-QAM signal modulated in optical intensity dimension is shaped geometrically and probabilistically. We used Hexagonal-QAM(HQAM) as a scheme of geometric constellation shaping to increase density of symbol and decrease peak symbol power. Also, probabilistic shaping is aided to HQAM model. With the mathematical analysis on IDI, symbol appearance probability is adjusted according to the intensity of the RF-QAM symbol. We verified the validity of the proposed technique with optical fiber transmission simulation.