Epigenetic reprogramming occurring during reproduction is crucial for both animal and plant development. Histone H3 Lys 4 trimethylation (H3K4me3) is an evolutionarily conserved epigenetic mark of transcriptional active euchromatin. While much has been learned in somatic cells, H3K4me3 deposition and function in gametophyte is poorly studied. Here, we demonstrate that SET DOMAIN GROUP2 (SDG2)-mediated H3K4me3 deposition participates in epigenetic reprogramming during Arabidopsis male gametogenesis. We show that loss of SDG2 barely affects meiosis and cell fate establishment of haploid cells. However, we found that SDG2 is critical for postmeiotic microspore development. Mitotic cell division progression is partly impaired in the loss-of-function sdg2-1 mutant, particularly at the second mitosis setting up the two sperm cells. We demonstrate that SDG2 is involved in promoting chromatin decondensation in the pollen vegetative nucleus, likely through its role in H3K4me3 deposition, which prevents ectopic heterochromatic H3K9me2 speckle formation. Moreover, we found that derepression of the LTR retrotransposon ATLANTYS1 is compromised in the vegetative cell of the sdg2-1 mutant pollen. Consistent with chromatin condensation and compromised transcription activity, pollen germination and pollen tube elongation, representing the key function of the vegetative cell in transporting sperm cells during fertilization, are inhibited in the sdg2-1 mutant. Taken together, we conclude that SDG2-mediated H3K4me3 is an essential epigenetic mark of the gametophyte chromatin landscape, playing critical roles in gamete mitotic cell cycle progression and pollen vegetative cell function during male gametogenesis and beyond.