Human epidermal growth factor receptor 2 (HER2) is a clinically validated target for breast cancer therapy. Previously, a drug-fused HER2-targeting affinity protein construct successfully extended the survival of mice bearing HER2-expressing xenografts. The aim of this study was to evaluate the influence of the number and positioning of the protein domains in the drug conjugate. Seven HER2-targeting affibody-based constructs, including one or two affibody molecules (Z) with or without an albumin-binding domain (ABD), namely Z, Z-ABD, ABD-Z, Z-Z, Z-Z-ABD, Z-ABD-Z, and ABD-Z-Z, were evaluated on their effects on cell growth, in vivo targeting, and biodistribution. The biodistribution study demonstrated that the monomeric constructs had longer blood retention and lower hepatic uptake than the dimeric ones. A dimeric construct, specifically ABD-Z-Z, could stimulate the proliferation of HER2 expressing SKOV-3 cells in vitro and the growth of tumors in vivo, whereas the monomeric construct Z-ABD could not. These two constructs demonstrated a therapeutic effect when coupled to mcDM1; however, the effect was more pronounced for the non-stimulating Z-ABD. The median survival of the mice treated with Z-ABD-mcDM1 was 63 days compared to the 37 days for those treated with ABD-Z-Z-mcDM1 or for the control animals. Domain permutation of an ABD-fused HER2-targeting affibody-based drug conjugate significantly influences tumor cell proliferation and therapy efficacy. The monomeric conjugate Z-ABD is the most promising format for targeted delivery of the cytotoxic drug DM1.