Acacia mangium is well known as a valuable commercial tree species in the Acacia genus. A. mangium was recently found to be self-incompatible (SI), but its SI mechanism is not clear, which has hindered the progress of genetic improvement of A. mangium with strong resistance. To confirm the SI type of A. mangium, pollen germination was observed via fluorescence microscopy at 0 h, 3 h, 6 h, 9 h, 12 h, and 24 h after self-pollination. We found over ninety percent of the pollen grains produced no pollen tube growth on the stigma/style. To further explore the SI molecular mechanism of A. mangium, tests of the transcriptome and metabolome were carried out after self-pollination. Observations of pollen germination after self-pollination using fluorescence microscopy suggested that the SI type of A. mangium is gametophytic self-incompatibility (GSI). A combined transcriptomic and metabolomic analysis showed that DEGs (differentially expressed genes) related to SI (6 S-glycoproteins, 93 F-box proteins, 69 26S proteasomes, 38 calcium-dependent protein kinases/calmodulin and 41 thioredoxin genes) were significantly enriched in six KEGG (sulfur metabolism, tyrosine metabolism, phenylalanine metabolism, butanoate metabolism, and valine, leucine, and isoleucine degradation). Further analysis of these six pathways revealed the enrichment of SI-related DEGs corresponding to succinate, methylmalonate, and 3-hydroxypropane. These three metabolites were significantly downregulated. The analysis of transcripts and metabolites suggested that transcripts of SI-related gene families (thioredoxin and F-box protein) were significantly upregulated under the regulation of transcription factors (TFs) after self-pollination, leading to a decrease in metabolites (such as succinate, methylmalonate, and 3-hydroxypropionate). We also further speculated that TFs (MYB, HB-HD-ZIP, AP2/ERF-ERF, and bZIP) and gene families (thioredoxin and F-box protein) were important factors related to the SI of A. mangium.