Interfacial layers are frequently used in organic solar cells performing various functions, including blocking surface recombination, improving selectivity of charge carrier extraction, modification of the work function of the contact materials and enhancing light absorption within the photoactive layer through an optical cavity effect. The aim of this work is to investigate the origin of performance enhancement of bulk heterojunction solar cells using various electron and hole interfacial layers with a particular focus on the improvement to the open circuit voltage (Voc). Solar cells using poly[N-9′-hepta-decanyl-2,7-carbazole-alt-5,5-(4′,7′-di-2-thienyl-2′,1′,3′benzothiadiazole)](PCDTBT) :)]:[6,6]-phenyl C70-butyric acid methyl ester (PC[70]BM) (1:4) active layers were prepared with a combination of polymeric, metal oxide and polyelectrolyte electron and or hole interfacial layers. Four device structures with i) no interfacial layers (reference); ii) only hole; iii) only electron; iv) both electron and hole interfacial layers were fabricated and compared using current-voltage, transient photovoltage and charge extraction measurements. The voltage gains (ΔVoc) at matched charge density due to work function modification (ΔVoc h or ΔVoc e) is distinguished from the increase in Voc due to increased charge carrier density due to longer charge carrier lifetime. At the hole contact, ΔVoc h was 0.21 V by using a PEDOT-PSS hole interfacial layer, while ΔVoc e was 0.29 V on the electron contact using a PEI-TiOx interfacial layer compared to reference devices. The electron lifetime also improved by orders of magnitude with the use of either electron or hole contact layers, contributing to a further 0.35-0.38 V increase in the open circuit voltage ((ΔVoc rec) due to increased charge density. The increased charge carrier lifetime is proposed to originate from the larger spatial separation of the electrons and holes in the device due to the increased internal field. Using both an electron and a hole interfacial layer didn't significantly increase the charge carrier lifetime compared to single interfacial layer devices, therefore the Voc didn't increase significantly. The findings presented clarify the role of interfacial layers in organic solar cells, and provides new insights into using time resolved charge extraction techniques to understand the influence of interfacial layers on the open circuit voltage.