Putting constraints on a possible Lorentz Invariance Violation (LIV) from astrophysical sources such as gamma-ray bursts (GRBs) is essential for finding evidences of new theories of quantum gravity (QG) that predict an energy-dependent speed of light. This search has its own difficulties, so usually, the effect of the cosmological model is understudied, with the default model being a fixed-parameters $\Lambda$CDM. In this work, we use various astrophysical datasets to study the effect of a number of dark energy models on LIV constraints. To this end, we combine two public time-delay GRB datasets with the supernovae Pantheon dataset, several measurements of angular baryonic acoustic oscillations (BAO), the cosmic microwave background (CMB) distance prior and an optional GRB or quasars dataset. For the LIV parameter $\alpha$, we find the expected from previous works average value of $\alpha \sim 4 \times 10^{-4}$, corresponding to $E_{QG}\ge 10^{17}$ GeV for both time-delay (TD) datasets, with the second one being more sensitive to the cosmological model. The cosmology results in a minimum 20\% deviation in our constraints on the energy. Interestingly, adding the TD points makes the DE models less-preferable statistically and shifts the value of the parameter $c/(H_0 r_d)$ down, making it smaller than the expected value. We observe that possible LIV measurements critically depend on the transparency of the assumptions behind the published data concerning cosmology, and taking this into account may be an important contribution in the case of possible detection.