The Large Hadron Collider (LHC) has provided direct evidence of Yukawa couplings between the third generation charged fermions and the 125 GeV Higgs boson. Whether the first two generation charged fermions arise from exactly the same mechanism becomes the next interesting question. Therefore, direct measurements of charm or muon Yukawa couplings will be crucial to answering this puzzle. The charm Yukawa measurement at the LHC suffers from severe QCD background and it is extremely difficult to reach the sensitivity. In this paper, we compare the potential of probing charm Yukawa coupling at the two proposed future "Higgs Factory" experiments, the Large Hadron electron Collider (LHeC) and Circular electron positron collider (CEPC). At the LHeC, Higgs bosons will be produced via weak boson fusion and the energetic forward jet may suppress the background significantly. However, due to huge γ − g scattering background, the potential of LHeC search is still limited. With a −80% polarized electron beam of 60 GeV, the signal significance can only reach 2σ for κ c ≃ 1.18 with a 3 ab −1 integrated luminosity. In contrast, measurement at the CEPC can reach 5.8σ for κ c ≃ 1 with a 2 ab −1 of data.