Detection of infected kernels is important for Fusarium head blight (FHB) prevention and product quality assurance in wheat. In this study, Raman spectroscopy (RS) and deep learning networks were used for the determination of FHB-infected wheat kernels. First, the RS spectra of healthy, mild, and severe infection kernels were measured and spectral changes and band attribution were analyzed. Then, the Inception network was improved by residual and channel attention modules to develop the recognition models of FHB infection. The Inception–attention network produced the best determination with accuracies in training set, validation set, and prediction set of 97.13%, 91.49%, and 93.62%, among all models. The average feature map of the channel clarified the important information in feature extraction, itself required to clarify the decision-making strategy. Overall, RS and the Inception–attention network provide a noninvasive, rapid, and accurate determination of FHB-infected wheat kernels and are expected to be applied to other pathogens or diseases in various crops.