A systematic study was carried out on 638 wheat and paddy grains (including fresh and stored samples) collected in 2021 from Shanghai, China, to identify the major mycobiota and their toxigenic abilities. A total of 349 fungi, namely, 252 Fusarium, 53 Aspergillus, and 44 Alternaria, were characterized by morphological and molecular identification. Fusarium and Aspergillus were more frequently isolated in paddy with Fusarium sambucinum species complex and Aspergillus section flavi as the predominant species, respectively. The genus Alternaria was the most frequently isolated fungal species in wheat. The toxin-producing potentials of the identified fungi were further evaluated in vitro. Deoxynevalenol (DON) was produced by 34.5% of Fusarium isolates and zearalenone (ZEN) was produced by 47.6% of them, and one isolate also processed the abilities for fumonisin B1 (FB1), B2 (FB2), and B3 (FB3) productions. Aflatoxin B1 (AFB1), B2 (AFB2), and G1 (AFG1) were only generated by Aspergillus section flavi, with the production rate of 65.5%, 27.6%, and 13.8%, respectively. Alternariol (AOH) was the most prevalent Alternaria toxin, which could be produced by 95.5% of the isolates, followed by alternariol monomethyl ether (AME) (72.7%), altenuene (ALT) (52.3%), tenuazonic acid (TeA) (45.5%), tentoxin (TEN) (29.5%), and altenusin (ALS) (4.5%). A combinational analysis of mycobiota and toxigenic ability allowed us to provide comprehensive information about the production mechanisms of mycotoxins in wheat and paddy in a specific geographic area, and will be helpful for developing efficient prevention and control programs.