YEATS (YAF9, ENL, AF9, TAF14, SAS5) family proteins recognize acylated histones and in turn regulate chromatin structure, gene transcription, and stress signaling. The chromosomal translocations of ENL and mixed lineage leukemia are considered oncogenic drivers in acute myeloid leukemia and acute lymphoid leukemia. However, known ENL YEATS domain inhibitors have failed to suppress the proliferation of 60 tested cancer cell lines. Herein, we identified four hits from the NMR fragment-based screening against the AF9 YEATS domain. Ten inhibitors of new chemotypes were then designed and synthesized guided by two complex structures and affinity assays. The complex structures revealed that these inhibitors formed an extra hydrogen bond to AF9, with respect to known ENL inhibitors. Furthermore, these inhibitors demonstrated antiproliferation activities in AF9-sensitive HGC-27 cells, which recapitulated the phenotype of the CRISPR studies against AF9. Our work will provide the basis for further structured-based optimization and reignite the campaign for potent AF9 YEATS inhibitors as a precise treatment for AF9-sensitive cancers.