In this study, two vital genes in the serine protease cascade, NlK-serpin and NlVenomase, were identified. Two genes show differential responses in brown planthopper (Nilaparvata lugens, BPH) carrying the insect fungal pathogen Metarhizium anisopliae YTTR and/or the plant viral pathogen rice ragged stunt virus (RRSV). RNA interference (RNAi) combined with/without M. anisopliae YTTR were used to explore the genes’ functions. And BPH mortality, the transcriptional response of four antimicrobial peptides (AMPs) genes, and the phenoloxidase (PO) activity were measured to unveil the underlying mechanism. Results showed that the coding sequences (CDSs) of NlK-serpin and NlVenomase contained 291-bp and 1,194-bp nucleotides, respectively. Phylogenetic analysis showed both genes have the closest kinship to Laodelphax striatellus. NlK-serpin is highly expressed in the adult and 3rd -instar stages and NlVenomase is highly expressed in 2nd and 3rd -instar nymphs. Both genes possess high expression in the fat body. Synthesized dsRNAs have strong interference efficiency against the expression of target genes. DsNlK-serpin injection significantly reduced BPHs survival rates in RRSV-free BPHs. However, when RRSV existed, the differences in survival rates among the three dsRNA treatments were insignificant. DsRNA combined with M. anisopliae YTTR produced a synergistic effect leading to higher BPHs mortality. Results of AMPs expression and PO activity indicated that NlK-serpin and NlVenomase play a vital role in BPH humoral immunity against M. anisopliae YTTR infection. And both genes participate in the process of RRSV-mediated immune responses of BPH to M. anisopliae YTTR infection by regulating AMPs expression and PO activity. Our study provided new targets for BPH control and laid a solid foundation for further research the interaction of “plant virus- insect vector- entomopathogenic fungus”.