Nontarget-site resistance (NTSR) is a complex multigenic trait that is associated with the potential mechanisms of herbicide resistance which pose a serious threat to global crop protection. However, the NTSR mechanisms of Alopecurus japonicus, a malignant weed infesting wheat fields, are less characterized. In this study, we used RNA-sequencing transcriptome and enzyme activity detection to investigate the NTSR mechanisms and candidate genes involved in fenoxaprop-P-ethyl (FE) in a previously identified resistant population compared to the sensitive population of A. japonicus. Transcriptome analysis identified nine upregulated genes, which were constitutively overexpressed and upregulated by FE application in the resistant population, and the results were validated using quantitative real-time PCR. These genes including one cytochrome P450 monooxygenase (P450) gene (CYP75B4), one ATP-binding cassette (ABC) transporter gene (ABCG36), one laccase (LAC) gene (LAC15), one 9-cis-epoxycarotenoid dioxygenase (NCED) gene (NCED5), two purple acid phosphatase (PAP) genes (PAP4, PAP15), one sucrose phosphate synthase (SPS) gene (SPS3), one protein related to disease resistance gene (RGA3) and one immune protein gene (R1B-17). The activity assay of LAC, NCED, PAP and SPS revealed that the activities of these enzymes in the resistant population were significantly higher than those in the sensitive population at 0 h and after FE application at 12 h, 24 h and 72 h. Nevertheless, whether LAC, NCED, PAP and SPS genes were involved in herbicide metabolism needs to be further validated. Our results revealed that CYP, ABC transporter and LAC genes may participate in A. japonicus resistance. These genes identified in the present study provide new insights into the resistance mechanism of weeds in response to herbicide. Our study also implies the complexity of the NTSR mechanisms of weeds.