We describe an efficient approach to modelling cavity quantum electrodynamics (QED) with a time-delayed coherent feedback using quantum trajectory simulations. An analytical set of equations is derived to exploit the advantages of trajectories in the presence of the non-Markovian dynamics, where adjustments to the standard stochastic dynamics are discussed. In the weak excitation regime, we first verify that our approach recovers known results obtained with other simulation methods and demonstrate how a coherent feedback loop can increase the photon lifetime in typical cavity-QED systems. We then explore the nonlinear few-photon regime of cavity-QED, under the restriction of at most one photon at a time in the feedback loop. In particular, we show how feedback affects the cavity photoluminescence (populations versus laser detuning), and describe how one must account for conditioning in the presence of feedback, specifically the system observables must be conditioned on no photon detections at the feedback output channel occurring.