Herein we report a comparative morphological analysis of the perylene diimide (PDI)- and fullerene-based organic solar cells (OSCs) to identify the factors responsible for low performance of PDI-based devices. A PDI derivative, bis-PDI, and a fullerene derivative, PC70BM, are mixed with an efficient polymer donor, PffBT4T-2OD. The large disparity in power conversion efficiencies (PCEs) of OSCs composed of PffBT4T-2OD:bis-PDI (PCE = 5.18%) and PffBT4T-2OD:PC70BM (PCE = 10.19%) observed are attributed to differences in the nanostructural motif of bulk heterojunction (BHJ) morphologies of these blend systems. The X-ray scattering and surface energy characterizations revealed that the structurally dissimilar bis-PDI and PC70BM molecules determine the variation in blend film morphologies, and in particular, the molecular packing features of the donor PffBT4T-2OD polymer. In addition, high-resolution transmission electron microscopy (HRTEM) images explore the BHJ morphologies and presence of longer polymer fibrils in PffBT4T-2OD:bis-PDI system, justifying the unbalanced charge transport and high hole mobility. The low performance of PffBT4T-2OD:bis-PDI devices was further investigated by studying charge carrier recombination dynamics by using light-intensity-dependent and transient photovoltage (TPV) experiments. Furthermore, the temperature-dependent experiments showed the photovoltaic properties, including charge recombination losses, are strongly affected by energetic disorder present in bis-PDI-based system.