The rhizosphere microbiome is a major determinant of plant health, which can interact with the host directly and indirectly to promote or suppress productivity. Oil palm is one of the world’s most important crops, constituting over a third of global vegetable oil production. Currently there is little understanding of the oil palm microbiome and its contribution to plant health and productivity, with existing knowledge based almost entirely on culture dependent studies. We investigated the diversity and composition of the oil palm fungal microbiome in the bulk soil, rhizosphere soil, and roots of 2-, 18-, and 35-year old plantations in Selangor, Malaysia. The fungal community showed substantial variation between the plantations, accounting for 19.7% of community composition, with compartment (root, rhizosphere soil, and bulk soil), and soil properties (pH, C, N, and P) contributing 6.5 and 7.2% of community variation, respectively. Rhizosphere soil and roots supported distinct communities compared to the bulk soil, with significant enrichment of Agaricomycetes, Glomeromycetes, and Lecanoromycetes in roots. Several putative plant pathogens were abundant in roots in all the plantations, including taxa related to Prospodicola mexicana and Pleurostoma sp. The mycorrhizal status and dependency of oil palm has yet to be established, and using 18S rRNA primers we found considerable between-site variation in Glomeromycotinian community composition, accounting for 31.2% of variation. There was evidence for the selection of Glomeromycotinian communities in oil palm roots in the older plantations but compartment had a weak effect on community composition, accounting for 3.9% of variation, while soil variables accounted for 9% of community variation. While diverse Mucoromycotinian fungi were detected, they showed very low abundance and diversity within roots compared to bulk soil, and were not closely related to taxa which have been linked to fine root endophyte mycorrhizal morphology. Many of the fungal sequences showed low similarity to established genera, indicating the presence of substantial novel diversity with significance for plant health within the oil palm microbiome.