Specific link between high fructose uptake and cancer development and progression highlighted fructose transporters as potential means to achieve GLUT-mediated discrimination between normal and cancer cells. The gained expression of fructose-specific transporter GLUT5 in various cancers offers a possibility for developing cancer-specific imaging and bioactive agents. Herein, we explore the feasibility of delivering a bioactive agent through cancer-relevant fructose-specific transporter GLUT5. We employed specific targeting of GLUT5 by 2,5-anhydro-D-mannitol and investigated several drug conjugates for their ability to induce cancer-specific cytotoxicity. The proof-of-concept analysis was carried out for conjugates of chlorambucil (CLB) in GLUT5-positive breast cancer cells and normal breast cells. The cytotoxicity of conjugates was assessed over 24 h and 48 h, and significant dependence between cancer-selectivity and conjugate size was observed. The differences were found to relate to the loss of GLUT5-mediated uptake upon increased conjugate size and hydrophobicity. The findings provide information on the substrate tolerance of GLUT5 and highlight the importance of maintaining appropriate hydrophilicity for GLUT-mediated delivery.