Lorentz Invariance Violation in Quantum Gravity (QG) models or a non-zero photon mass, mγ, would lead to an energy-dependent propagation speed for photons, such that photons of different energies from a distant source would arrive at different times, even if they were emitted simultaneously. By developing source-by-source, Monte Carlo-based forward models for such time delays from Gamma Ray Bursts, and marginalising over empirical noise models describing other contributions to the time delay, we derive constraints on mγ and the QG length scale, QG, using spectral lag data from the BATSE satellite. We find mγ < 4.0 × 10 −5 h eV/c 2 and QG < 5.3 × 10 −18 h GeV −1 at 95% confidence, and demonstrate that these constraints are robust to the choice of noise model. The QG constraint is among the tightest from studies which consider multiple Gamma Ray Bursts and the constraint on mγ, although weaker than from using radio data, provides an independent constraint which is less sensitive to the effects of dispersion by electrons.