Most toxic agents currently used for chemotherapy show a narrow therapeutic window, because of their inability to distinguish between healthy and cancer cells. Targeted drug delivery offers the possibility to overcome this issue by selectively addressing structures on the surface of cancer cells, therefore reducing undesired side effects. In this broad field, peptide-drug conjugates linked by intracellular cleavable structures have evolved as highly promising agents. They can specifically deliver toxophores to tumor cells by targeting distinct receptors overexpressed in cancer. In this review, we focus on these compounds and describe important factors to develop a highly efficient peptide-drug conjugate. The necessary properties of tumor-targeting peptides are described, and the different options for cleavable linkers used to connect toxic agents and peptides are discussed, and synthetic considerations for the introduction of these structures are reported. Furthermore, recent examples and current developments of peptide-drug conjugates are critically evaluated with a special focus on the applied linker structures and their future use in cancer therapy.