Antibody-drug conjugates (ADCs) are currently used for the targeted delivery of drugs to diseased cells, but intracellular drug delivery and therefore efficacy may be suboptimal because of the large size, slow internalization and ineffective intracellular trafficking of the antibody. Using a phage display method selecting internalizing phages only, we developed internalizing single domain antibodies (sdAbs) with high binding affinity to rat PDGFRβ, a receptor involved in different types of diseases. We demonstrate that these constructs have different characteristics with respect to internalization rates but all traffic to lysosomes. To compare their efficacy in targeted drug delivery, we conjugated the sdAbs to a cytotoxic drug. The conjugates showed improved cytotoxicity correlating to their internalization speed. The efficacy of the conjugates was inhibited in the presence of vacuolin-1, an inhibitor of lysosomal maturation, suggesting lysosomal trafficking is needed for efficient drug release. In conclusion, sdAb constructs with different internalization rates can be designed against the same target, and sdAbs with a high internalization rate induce more cell killing than sdAbs with a lower internalization rate in vitro. Even though the overall efficacy should also be tested in vivo, sdAbs are particularly interesting formats to be explored to obtain different internalization rates.