The regulation of gene expression is a fundamental process enabling cells to respond to internal and external stimuli or to execute developmental programs. Changes in gene expression are highly dynamic and depend on many intrinsic and extrinsic factors. In this review, we highlight the dynamic nature of transient gene expression changes to better understand cell physiology and development in general. We will start by comparing recent in vivo procedures to capture gene expression in real time. Intrinsic factors modulating gene expression dynamics will then be discussed, focusing on chromatin modifications. Furthermore, we will dissect how cell physiology or age impacts on dynamic gene regulation and especially discuss molecular insights into acquired transcriptional memory. Finally, this review will give an update on the mechanisms of heterogeneous gene expression among genetically identical individual cells. We will mainly focus on state-of-the-art developments in the yeast model but also cover higher eukaryotic systems.