Dark matter (DM) coupled to light mediators has been invoked to resolve the putative discrepancies between collisionless cold DM and galactic structure observations. However, γ-ray searches and the CMB strongly constrain such scenarios. To ease the tension, we consider asymmetric DM. We show that, contrary to the common lore, detectable annihilations occur even for large asymmetries, and derive bounds from the CMB, γ-ray, neutrino and antiproton searches. We then identify the viable space for self-interacting DM. Direct detection does not exclude this scenario, but provides a way to test it.