Mining activities, ore concentrations, and transport processes generate large amounts of pollutants, including hazardous elements, which are released into the environment. This work presents the results of experimental research aimed at evaluating the environmental risks of soil and plant contamination in two magnesite mining and processing areas in the Slovak Republic, and assesses the phytoremediation potential of dominant plant species. Eleven potentially toxic elements in the soil were investigated using X-ray fluorescence spectrometry (Cd, Pb, Cr, Zn, Cu, As, Ni, Mn, Mg, Fe) and atomic absorption spectrometry (Hg). In plants, potentially toxic elements were investigated using inductively coupled plasma mass spectrometry (Cu, As, Cd, Pb) and inductively coupled plasma atomic emission spectrometry (Cr, Zn, Mn, Mg). Selected soil parameters (pH, redox potential, and soil organic matter) were also investigated. Soil contamination was evaluated using environmental indices (geoaccumulation index—Igeo, enrichment factor—EF, contamination factor —Cf, degree of contamination—Cd). The phytoremediation potential of plants was evaluated using the bioconcentration factor (BCF) and the translocation factor (TF). The soil reaction in the studied areas indicated a strong alkalization of the soil. The soils in Jelšava-Lubeník were significantly contaminated with Cr, As, Mn, and Mg. The most significant enrichment based on the average values of EF was found to be in the order of Cd > Mg > Zn > Cu > As > Cr > Ni > Pb > Fe > Hg > Mn. The observed values of Cf and Cd indicated a high degree of soil contamination. In Košice, the soils were found to be significantly contaminated with Cr, Mn, Mg, and Ni. The most significant enrichment was found in the order of Cd > Mn > Ni > Pb > Zn > Mg > Cu > As > Fe > Cr > Hg. Very high Cf was found for Pb and Cr. The results of correlation and hierarchical cluster analyses suggest a similar origin of pollutants caused by significant anthropogenic interventions due to magnesite mining and processing. The investigated dominant plant species, Phragmites australis, Agrostis stolonifera, Elytrigia repens, and Taraxacum officinale are able to accumulate high concentrations of the monitored potentially toxic elements without more serious load or damage. The results of BCF and TF confirmed that P. australis and T. officinale appeared to be suitable accumulators in the phytoextraction process. In the case of E. repens and A. stolonifera it was confirmed that they accumulate and immobilize high concentrations of potentially toxic elements, especially in the roots, establishing the suitability of their use in phytostabilization processes.