TRPV1 and TRPV4, members of the transient receptor potential vanilloid family, are multimodal ion channels activated by various stimuli, including temperature and chemicals. It has been demonstrated that TRPV channels function as tetramers; however, the dynamics of the diffusion, oligomerization, and endocytosis of these channels in living cells are unclear. Here we undertook single-molecule time-lapse imaging of TRPV1 and TRPV4 in HEK 293 cells. Differences were observed between TRPV1 and TRPV4 before and after agonist stimulation. In the resting state, TRPV4 was more likely to form higher-order oligomers within immobile membrane domains than TRPV1. TRPV1 became immobile after capsaicin stimulation, followed by its gradual endocytosis. In contrast, TRPV4 was rapidly internalized upon stimulation with GSK1016790A. The selective loss of immobile higher-order oligomers from the cell surface through endocytosis increased the proportion of the fast-diffusing state for both subtypes. With the increase in the fast state, the association rate constants of TRPV1 and TRPV4 increased, regenerating the higher-order oligomers. Our results provide a possible mechanism for the different rates of endocytosis of TRPV1 and TRPV4 based on the spatial organization of the higher-order structures of the two TRPV channels.