We develop a method to simulate the excitonic dynamics of realistic photosynthetic light harvesting systems including non-Markovian coupling to phonon degrees of freedom, under excitation by N-photon Fock state pulses. This method combines the input-output formalism and the hierarchical equations of motion (HEOM) formalism into a double hierarchy of coupled linear equations in density matrices. We show analytically that in a density matrix description, under weak field excitation relevant to natural photosynthesis conditions, an N-photon Fock state input and a corresponding coherent state input give rise to equal density matrices in the excited manifold. However, an important difference is that an Nphoton Fock state input has no off-diagonal coherence between the ground and excited subspaces, in contrast with the coherences created by a coherent state input. We derive expressions for the probability to absorb a single Fock state photon, with or without the influence of phonons. For short pulses (or equivalently, wide bandwidth pulses), we show that the absorption probability has a universal behavior that depends only upon a system-dependent effective energy spread parameter ∆ and an exciton-light coupling constant Γ. This holds for a broad range of chromophore systems and for a variety of pulse shapes. We also analyse the absorption probability in the opposite long pulses (narrow bandwidth) regime. We then derive an expression for the long time emission rate in the presence of phonons and use it to study the difference between collective versus independent emission. Finally, we present a numerical simulation for the LHCII monomer (14-mer) system under single photon excitation that illustrates the use of the double hierarchy for calculation of Fock state excitation of a light harvesting system including chromophore coupling to a non-Markovian phonon bath.