The two leading hypotheses for the Galactic Center Excess (GCE) in the Fermi data are an unresolved population of faint millisecond pulsars (MSPs) and dark-matter (DM) annihilation. The dichotomy between these explanations is typically reflected by modeling them as two separate emission components. However, point-sources (PSs) such as MSPs become statistically degenerate with smooth Poisson emission in the ultra-faint limit (formally where each source is expected to contribute much less than one photon on average), leading to an ambiguity that can render questions such as whether the emission is PS-like or Poissonian in nature ill-defined. We present a conceptually new approach that describes the PS and Poisson emission in a unified manner and only afterwards derives constraints on the Poissonian component from the so obtained results. For the implementation of this approach, we leverage deep learning techniques, centered around a neural network-based method for histogram regression that expresses uncertainties in terms of quantiles. We demonstrate that our method is robust against a number of systematics that have plagued previous approaches, in particular DM / PS misattribution. In the Fermi data, we find a faint GCE described by a median source-count distribution (SCD) peaked at a flux of ∼ 4 × 10 −11 counts cm −2 s −1 (corresponding to ∼ 3 − 4 expected counts per PS), which would require N ∼ O(10 4 ) sources to explain the entire excess (median value N = 29,300 across the sky). Although faint, this SCD allows us to derive the constraint ηP ≤ 66% for the Poissonian fraction of the GCE flux ηP at 95% confidence, suggesting that a substantial amount of the GCE flux is due to PSs.