Imaging and chromosome conformation capture studies have revealed several layers of chromosome organization, including segregation into megabase-large active and inactive compartments, and partitioning into sub-megabase domains (TADs). Yet, it remains unclear how these layers of organization form, interact with one another and impact genome functions. Here, we show that deletion of the cohesin-loading factor Nipbl, in mouse liver, leads to a dramatic reorganization of chromosomal folding. TADs and associated peaks vanish globally, even in the absence of transcriptional changes. In contrast, compartmental segregation is preserved and even reinforced. Strikingly, the disappearance of TADs unmasks a finer compartment structure that accurately reflects the underlying epigenetic landscape. These observations demonstrate that the 3D organization of the genome results from the interplay of two independent mechanisms: 1) cohesin-independent segregation of the genome into fine-scale compartments, defined by chromatin state; 2) cohesin-dependent formation of TADs, possibly by loop extrusion, which contributes to guide distant enhancers to their target genes.