The genome of eukaryotes is highly organized within the cell nucleus, this organization per se elicits gene regulation and favors other mechanisms like cell memory throughout histones and their post-translational modifications. In highly specialized cells, like sperm, the genome is mostly organized by protamines, yet a significant portion of it remains organized by histones. This protamine-histone-DNA organization, known as sperm epigenome, is established during spermiogenesis. Specific histones and their posttranslational modifications are retained at specific genomic sites and during embryo development these sites recapitulate their histone profile that harbored in the sperm nucleus. It is known that histones are the conduit of epigenetic memory from cell to cell, hence histones in the sperm epigenome may have a role in transmitting epigenetic memory from the sperm to the embryo. However, the exact function and mechanism of histone retention remains elusive. During spermatogenesis, most of the histones that organize the genome are replaced by protamines and their retention at specific regions may be deeply intertwined with the eviction and replacement mechanism. In this review we will cover some relevant aspects of histone replacement that in turn may help us to contextualize histone retention. In the end, we focus on the architectonical protein CTCF that is, so far, the only factor that has been directly linked to the histone retention process.