Background: Nutritional food quality plays a crucial role in maintaining human health. However, food and drinking water, along with occupational exposure, are the main routes of exposure to toxic elements for humans. The main aim of this study was to determine the content of As, Cd, Pb and Hg in naturally gluten-free grains and products made from buckwheat, millet, maize, quinoa and oat. The safety of consumption of the products tested was also assessed. Methods: The contents of As, Cd and Pb were determined using inductively coupled plasma mass spectrometry (ICP-MS). To measure Hg, an atomic absorption spectrometry method (AAS) with the amalgamation technique was applied. To assess the level of consumption of the tested products, an online survey was conducted. To estimate health risk, three indicators were used: the target hazard quotient (THQ), cancer risk (CR) and hazard index (HI). The research material obtained 242 different samples without replications. Results: The highest average content of As, Cd, Pb and Hg were observed for the following groups of products: oat (10.19 µg/kg), buckwheat (48.35 µg/kg), millet (74.52 µg/kg) and buckwheat (1.37 µg/kg), respectively. For six samples, exceedance of established limits was found—three in the case of Cd and three of Pb. Due to the lack of established limits, As and Hg content of the tested products was not compared. Generally, no increased health risks were identified. Conclusions: Based on the obtained results, the consumption of gluten-free cereals and pseudocereals available on the Polish market seems to be safe. However, there is a great need to establish maximum levels of the toxic elements, especially As and Hg in cereal products in European legislation.