The penitremane and janthitremane families of indole-diterpenes are abundant natural products synthesized by Penicillium crustosum and P. janthinellum. Using a combination of PCR, cosmid library screening, and Illumina sequencing we have identified gene clusters encoding enzymes for the synthesis of these compounds. Targeted deletion of penP in P. crustosum abolished the synthesis of penitrems A, B, D, E, and F, and led to accumulation of paspaline, a key intermediate for paxilline biosynthesis in P. paxilli. Similarly, deletion of janP and janD in P. janthinellum abolished the synthesis of prenyl-elaborated indole-diterpenes, and led to accumulation in the latter of 13-desoxypaxilline, a key intermediate for the synthesis of the structurally related aflatremanes synthesized by Aspergillus flavus. This study helps resolve the genetic basis for the complexity of indole-diterpene natural products found within the Penicillium and Aspergillus species. All indole-diterpene gene clusters identified to date have a core set of genes for the synthesis of paspaline and a suite of genes encoding multi-functional cytochrome P450 monooxygenases, FAD dependent monooxygenases, and prenyl transferases that catalyse various regio- and stereo- specific oxidations that give rise to the diversity of indole-diterpene products synthesized by this group of fungi.