The transient receptor potential vanilloid 1 (TRPV1) channel belongs to the transient receptor potential channel superfamily and participates in many physiological processes. TRPV1 modulators (both agonists and antagonists) can effectively inhibit pain caused by various factors and have curative effects in various diseases, such as itch, cancer, and cardiovascular diseases. Therefore, the development of TRPV1 channel modulators is of great importance. In this study, the structure‐based virtual screening and ligand‐based virtual screening methods were used to screen compound databases respectively. In the structure‐based virtual screening route, a full‐length human TRPV1 protein was first constructed, three molecular docking methods with different precisions were performed based on the hTRPV1 structure, and a machine learning‐based rescoring model by the XGBoost algorithm was constructed to enrich active compounds. In the ligand‐based virtual screening route, the ROCS program was used for 3D shape similarity searching and the EON program was used for electrostatic similarity searching. Final 77 compounds were selected from two routes for in vitro assays. The results showed that 8 of them were identified as active compounds, including three hits with IC50 values close to capsazepine. In addition, one hit is a partial agonist with both agonistic and antagonistic activity. The mechanisms of some active compounds were investigated by molecular dynamics simulation, which explained their agonism or antagonism.