Heavy ecstasy use has been associated with neurocognitive deficits in various behavioral and brain imaging studies. However, this association is not conclusive owing to the unavoidable confounding factor of polysubstance use. The present study, as part of the Netherlands XTC Toxicity study, investigated specific effects of ecstasy on working memory, attention, and associative memory, using functional magnetic resonance imaging (fMRI). A large sample (n ¼ 71) was carefully composed based on variation in the amount and type of drugs that were used. The sample included 33 heavy ecstasy users (mean 322 pills lifetime). Neurocognitive brain function in three domains: working memory, attention, and associative memory, was assessed with performance measures and fMRI. Independent effects of the use of ecstasy, amphetamine, cocaine, cannabis, alcohol, tobacco, and of gender and IQ were assessed and separated by means of multiple regression analyses. Use of ecstasy had no effect on working memory and attention, but drug use was associated with reduced associative memory performance. Multiple regression analysis showed that associative memory performance was affected by amphetamine much more than by ecstasy. Both drugs affected associative memory-related brain activity, but the effects were consistently in opposite directions, suggesting that different mechanisms are at play. This could be related to the different neurotransmitter systems these drugs predominantly act upon, that is, serotonin (ecstasy) vs dopamine (amphetamine) systems.