M-QAM and MPSK are important modulation techniques. They are used to modulate systems. M-QAM is a popular scheme in high rate and high bandwidth efficiency systems while QAM is a combination of amplitude and phase modulation. Mathematically the M-QAM is described by the combined amplitude and phase modulation results. MPSK stands for Mary Phase Shift Keying and is a digital modulation process which conveys data by changing the modulating phase of the constant frequency. AWGN or Additive White Gaussian Noise is a basic noise model used to Mimic random processes that occur in nature. This research creates an optimization model for the M-QAM and MPSK Modulation in the AWGN Channel using MATLAB Simulink. Comparison of the two techniques was also performed.