Restriction factors are structurally and functionally diverse cellular proteins that constitute a first line of defense against viral pathogens. Exceptions exist, but typically these proteins are upregulated by interferons (IFNs), target viral components, and are rapidly evolving due to the continuous virus–host arms race. Restriction factors may target HIV replication at essentially each step of the retroviral replication cycle, and the suppression of viral transcription and the degradation of viral RNA transcripts are emerging as major innate immune defense mechanisms. Recent data show that some antiviral factors, such as the tripartite motif-containing protein 22 (TRIM22) and the γ-IFN-inducible protein 16 (IFI16), do not target HIV-1 itself but limit the availability of the cellular transcription factor specificity protein 1 (Sp1), which is critical for effective viral gene expression. In addition, several RNA-interacting cellular factors including RNAse L, the NEDD4-binding protein 1 (N4BP1), and the zinc finger antiviral protein (ZAP) have been identified as important immune effectors against HIV-1 that may be involved in the maintenance of the latent viral reservoirs, representing the major obstacle against viral elimination and cure. Here, we review recent findings on specific cellular antiviral factors targeting HIV-1 transcription or viral RNA transcripts and discuss their potential role in viral latency.