A polynomial time ultrapower is a structure given by the set of polynomial time computable functions modulo some ultrafilter. They model the universal theory ∀PV of all polynomial time functions. Generalizing a theorem of Hirschfeld (1975), we show that every countable model of ∀PV is isomorphic to an existentially closed substructure of a polynomial time ultrapower. Moreover, one can take a substructure of a special form, namely a limit polynomial time ultrapower in the classical sense of Keisler (1963). Using a polynomial time ultrapower over a nonstandard Herbrand saturated model of ∀PV we show that ∀PV is consistent with a formal statement of a polynomial size circuit lower bound for a polynomial time computable function. This improves upon a recent result of Krajíček and Oliveira (2017). * Based on the first author's Master Thesis [4] written under the supervision of the second author. 1 All relevant technical notions will be defined precisely later. This is a post-peer-review, pre-copyedit version of an article published in Archive for mathematical logic.