It has been reported in the literature that the survival probability P (t) near an exceptional point where two eigenstates coalesce should generally exhibit an evolution P (t) ∼ t 2 e −Γt , in which Γ is the decay rate of the coalesced eigenstate; this has been verified in a microwave billiard experiment [B. Dietz, et al., Phys. Rev. E 75, 027201 (2007)]. However, the heuristic effective Hamiltonian that is usually employed to obtain this result ignores the possible influence of the continuum threshold on the dynamics. By contrast, in this work we employ an analytical approach starting from the microscopic Hamiltonian representing two simple models in order to show that the continuum threshold has a strong influence on the dynamics near exceptional points in a variety of circumstances. To report our results, we divide the exceptional points in Hermitian open quantum systems into two cases: at an EP2A two virtual bound states coalesce before forming a resonance, anti-resonance pair with complex conjugate eigenvalues, while at an EP2B two resonances coalesce before forming two different resonances. For the EP2B, which is the case studied in the microwave billiard experiment, we verify the survival probability exhibits the previously reported modified exponential decay on intermediate timescales, but this is replaced with an inverse power law on very long timescales. Meanwhile, for the EP2A the influence from the continuum threshold is so strong that the evolution is non-exponential on all timescales and the heuristic approach fails completely. When the EP2A appears very near the threshold we obtain the novel evolution P (t) ∼ 1 − C1 √ t on intermediate timescales, while further away the parabolic decay (Zeno dynamics) on short timescales is enhanced.