The performance of photovoltaic devices based on blends of conjugated polymers with non-fullerene acceptors depends upon the phase behaviour and microstructure of the binary, which in turn depends on the chemical structures of the molecular components and the blend composition. We investigate the correlation between molecular structure, composition, phase behaviour and device performance of a model system comprising semi-crystalline poly-3-hexylthiophene (P3HT) as the donor polymer and three non-fullerene acceptors, two of which (O-IDTBR/EH-IDTBR) have a planar core with different side-chains, and one (O-IDFBR) has a twisted core. We combine differential scanning calorimetry with optical measurements including UV-Vis, photoluminescence, spectroscopic ellipsometry and Raman, and photovoltaic device performance measurements, all at varying blend composition. For P3HT:IDTBR blends, the crystallinity of polymer and acceptor are preserved over a wide composition range and the blend displays a eutectic phase behaviour, with the optimum solar cell composition lying close to the eutectic. For P3HT:IDFBR blends, increasing acceptor content disrupts the polymer crystallinity, and the optimum device composition appears to be limited by polymer connectivity rather than being linked to the eutectic. The optical probes allow us to probe both the crystalline and amorphous phases, clearly revealing the compositions at which component mixing disrupts crystallinity.