A new method to constrain the dark matter annihilation cross section by taking the cross correlation of γ-ray diffuse maps and galaxies is proposed. As a result, the statistical power of constraining the annihilation cross section is proportional to the galaxy mass and the inverse square of the distance to the galaxy. Therefore, for analyses using dwarf galaxies, the measurement of the distance to the dwarf galaxies is crucial for stringent constraints. However, measuring the distance to galaxies is, in general observationally expensive, particularly for faint or diffuse galaxies. Under the situation of stacking N galaxies, we show that the distance to the individual galaxy measurement is not required, but the overall distance distribution is sufficient for the cross section constraints. Further, based on real Large Area Telescope (LAT) data, we show that the effect of covariance between two galaxies located closely, typically comparable with the point spread function size of the Fermi-LAT, is negligibly small. We can independently stack the likelihood for the N galaxies, which dramatically reduces the computation costs.By using actual datasets of the LAT γ-ray sky and ∼800 faint objects discovered by Hyper Suprime-Cam, we find that the upper limit on the annihilation cross-section scales with 1/N . Therefore, it can be one of the most powerful and robust probe of the annihilation signal to stack more than 10 5 galaxies readily available with the Legacy Survey of Space and Time.