In the mean-field framework, by variational analysis and numerical simulation, we investigate the effect of anharmonic trap and atomic interaction on the ground-state phases of a spin-orbit (SO) coupled Bose-Einstein condensates (BECs) in the harmonic plus quartic potential. Then, the Gaussian wave function is selected to predict the analytical conditions of the phase transition boundary of the SO coupled BECs by using the variational method. We found that the anharmonicity of the external potential induces the SO coupled BECs to undergo a phase transition between the zero-momentum phase and plane-wave phase, which is more pronounced in the cases of weak harmonic potential or strong interspecies interaction. Since the potential energy of the system modified by anharmonicity competes with other energies of the system, the anharmonicity changes the critical SO coupling strength and Raman coupling strength when the phase transition occurs. At the same time, the critical anharmonic coefficients are also affected by interspecies interaction and harmonic potential. Finally, the correctness of the theoretical results is verified by the numerical simulation of the Gross-Pitaevskii (GP) equation.