We prove that the convergence of the real and imaginary parts of the logarithm of the characteristic polynomial of unitary Brownian motion toward Gaussian free fields on the cylinder, as the matrix dimension goes to infinity, holds in certain suitable Sobolev spaces, whose regularity we prove to be optimal. Our result can be seen as the natural dynamical analogue to the stationary result for a fixed time by Hughes, Keating and O’Connell [1]. Further our result is related to the work of Spohn [2], from which the identification of the above limit as the Gaussian free field first followed, albeit in a different function space.