SPICA, the cryogenic infrared space telescope recently pre-selected for a "Phase A" concept study as one of the three remaining candidates for ESA's fifth medium class (M5) mission, is foreseen to include a far-infrared polarimetric imager (SPICA-POL, now called B-BOP), which would offer a unique opportunity to resolve major issues in our understanding of the nearby, cold magnetized Universe. This paper presents an overview of the main science drivers for B-BOP, including high dynamic range polarimetric imaging of the cold interstellar medium (ISM) in both our Milky Way and nearby galaxies. Thanks to a cooled telescope, B-BOP will deliver wide-field 100-350 µm images of linearly polarized dust emission in Stokes Q and U with a resolution, signal-to-noise ratio, and both intensity and spatial dynamic ranges comparable to those achieved by Herschel images of the cold ISM in total intensity (Stokes I). The B-BOP 200 µm images will also have a factor ∼ 30 higher resolution than Planck polarization data. This will make B-BOP a unique tool for characterizing the statistical properties of the magnetized interstellar medium and probing the role of magnetic fields in the formation and evolution of the interstellar web of dusty molecular filaments giving birth to most stars in our Galaxy. B-BOP will also be a powerful instrument for studying the magnetism of nearby galaxies and testing galactic dynamo models, constraining the physics of dust grain alignment, informing the problem of the interaction of cosmic rays with molecular clouds, tracing magnetic fields in the inner layers of protoplanetary disks, and monitoring accretion bursts in embedded protostars.