The three-dimensional (3D) genome organization and its role in biological activities have been investigated for over a decade in the field of cell biology. Recent studies using live-imaging and polymer simulation have suggested that the higher-order chromatin structures are dynamic; the stochastic fluctuations of nucleosomes and genomic loci cannot be captured by bulk-based chromosome conformation capture techniques (Hi-C). In this review, we focus on the physical nature of the 3D genome architecture. We first describe how to decode bulk Hi-C data with polymer modeling. We then introduce our recently developed PHi-C method, a computational tool for modeling the fluctuations of the 3D genome organization in the presence of stochastic thermal noise. We also present another new method that analyzes the dynamic rheology property (represented as microrheology spectra) as a measure of the flexibility and rigidity of genomic regions over time. By applying these methods to real Hi-C data, we highlighted a temporal hierarchy embedded in the 3D genome organization; chromatin interaction boundaries are more rigid than the boundary interior, while functional domains emerge as dynamic fluctuations within a particular time interval. Our methods may bridge the gap between live-cell imaging and Hi-C data and elucidate the nature of the dynamic 3D genome organization.