Due to the great potential of GaN based devices, the analysis of the growth of crack-free GaN with high quality has always been a research hotspot. In this paper, two methods for improving the property of the GaN epitaxial layer on Si (111) substrate are researched. Sample A, as a reference, only has an AlN buffer between the Si substrate and the epitaxy. In the following two samples, a GaN transition layer (sample B) and an AlGaN buffer (sample C) are grown on the AlN buffer separately. Both methods improve the quality of GaN. Meanwhile, using the second method, the residual tensile thermal stress decreases. To further study the impact of the two introduced layers, we investigate the stress condition of GaN epitaxial layer by Raman spectrum. According to the Raman spectrum, the calculated residual stress in the GaN epitaxial layer is approximately 0.72 GPa for sample B and 0.42 GPa for sample C. The photoluminescence property of GaN epitaxy is also investigated by room temperature PL spectrum.