The widespread usage of granite in the building sector motivated us to conduct this research and examine the material’s sustainability in terms of the investigated characteristics. The purpose of this paper is to discuss the statistical analysis results for the mineralogical impact on radiological hazards indices, such as the equivalent of radium, absorbed gamma dose rate, annual effective dose, internal and external hazard indices, as well as the gamma-ray index, that were calculated to estimate the environmental risks associated with these granites used as building materials, to protect the public from excessive radioactivity exposure. We focused primarily on statistical significance at a 95% confidence level. We employed a non-parametric test (Kruskal–Wallis Test) rather than a one-way ANOVA, to determine the statistical significance of the samples due to the lack of homogeneity or normality among them. To assess the difference between the samples, we used the Mann–Whitney Test on each pair of samples. Additionally, Pearson correlation coefficients for all the mineralogical results are computed. The presence of K-rich minerals (Kefeldspars, biotite) and accessories such as uranophane, uranothorite, allanite, xenotime, fergusonite, aeschynite, zircon, cassiterite, apatite, and sphene, which are mostly found in granitic rocks, determines the level of natural radioactivity of the investigated granites. Most of the rock samples analyzed have indicators of radioactive dangers that are within the acceptable level range, indicating that they are suitable for use as building materials. On the other hand, some samples have environmental criteria that are higher than international standards, indicating that they are unsuitable for use as construction materials.