Coal as the source of critical elements has attracted much attention and the enrichment mechanisms are of significant importance. This paper has an opportunity to investigate the mineralogical and geochemical characteristics of the Permian and Jurassic bituminous coals and associated non-coals from two underground coal mines in the Shanbei Coalfield (Northeast Ordos basin), Shaanxi Province, North China, based on the analysis of X-ray diffraction (XRD), inductively coupled plasma atomic-emission spectrometry (ICP-AES/MS), and scanning electron microscopy-energy-dispersive X-ray spectroscopy (SEM-EDS). The Jurassic and Permian coals have similar chemical features excluding ash yield, which is significantly higher in the Permian coals. Major mineral matters in the Jurassic coals are quartz, kaolinite, and calcite. By contrast, mineral assemblages of the Permian coals are dominated by kaolinite; and apatite occurring in the middle section’s partings. The Jurassic coals are only enriched in B, whereas the Permian coals are enriched in some trace elements (e.g., Nb, Ta, Th, and REY). Boron has a mixed inorganic and organic association which may be absorbed by organic matter from fluid (or groundwater) or inherited from coal-forming plants. Additionally, climatic variation also plays an important role. As for the Permian coals, kaolinite and apatite as the major carriers of elevated elements; the former were derived from the sedimentary source region (the Yinshan Oldland and the Benxi formation) and later precipitated from Ca-, and P-rich solutions. We deduced that those elevated elements may be controlled by the source rock and diagenetic fluid input. The findings of this work offered new data to figure out the mechanism of trace element enrichment of coal in the Ordos basin.