The human plasma metabolome contains thousands of metabolites that are dependent on an individual’s diet, genetics and gut microbiome, and on their interactions. By assessing 1,183 plasma metabolites in 1,368 extensively phenotyped individuals from the LifeLines-DEEP and GoNL cohorts, we quantified the proportion of inter-individual variation of the plasma metabolome explained by different factors and characterized 605 metabolites that are dominantly driven by diet, 85 driven by the gut microbiome, and 56 driven by genetics. We also developed a model that reflects the quality of an individual’s diet using 76 mostly diet-driven metabolites. By applying Mendelian randomization (MR) and mediation analyses, we reveal putative causal relationships between diet, gut microbiome and metabolites. For example, MR supports the causal effect of Eubacterium rectale in decreasing plasma levels of toxic p-cresol and p-cresol sulfate, two metabolites related to cardiometabolic risk and chronic kidney disease. We further followed-up the plasma metabolome profile in 311 individuals after 4 years and observe that the metabolites with more variance explained by genetics, microbiome or diet are also more temporarily stable. Altogether, our characterization of the dominant drivers of plasma metabolites can help in the design of therapeutic approaches that target the diet, human genome, or gut microbiome to drive a healthy metabolome.