The “Baganuur” lignite coal mine is one of the biggest open cast mines in Mongolia. However, there is a huge challenge in managing the stability of its internal dump, which prevents the proper operation of the mine and has an impact on the economy. To solve the internal dump slope stability problem, this study focused on incorporating the inherent mechanical properties of the rock material to build numerical models of the internal dump. By applying two software programs from Rocscience (Phase2 and Slide) and four different methods, the finite element method, the Bishop method, the Janbu simplified method, and the Spencer simplified method, the current and improved internal dump parameters were numerically simulated and analyzed. Based on the properties of the rock, the LEM and FEM were used to determine the parameters that could have an impact on the stability of the internal waste dump. The impacts of the internal dump height, dip angle, and safety berm on these parameters were studied. This study covers several analytical methods for calculating safety factors. Based on the results of the numerical simulation, it is determined that it is possible to increase the internal dump capacity by approximately 56% at a 50 m height and 28° dip angle and using a 15 m safety berm. Under similar conditions, this study presents an optimum SRF at 40 m height, 28° dip angle, and 5 m safety berm. Based on the numerical models, it is found that changes in the dip angle have a greater impact than changes in the dump height on the slope stability of an internal dump.