Several thousand sex-differential distal enhancers have been identified in mouse liver; however, their links to sex-biased genes and the impact of any sex-differences in nuclear organization and chromatin interactions are unknown. To address these issues, we first characterized 1,847 mouse liver genomic regions showing significant sex differential occupancy by cohesin and CTCF, two key 3D nuclear organizing factors. These sex-differential binding sites were primarily distal to sex-biased genes but rarely generated sex-differential TAD (topologically associating domain) or intra-TAD loop anchors, and were sometimes found in TADs without sex-biased genes. A substantial subset of sex-biased cohesin-non-CTCF binding sites, but not sex-biased cohesin-and-CTCF binding sites, overlapped sex-biased enhancers. Cohesin depletion reduced the expression of male-biased genes with distal, but not proximal, sex-biased enhancers by >10-fold, implicating cohesin in long-range enhancer interactions regulating sex-biased genes. Using circularized chromosome conformation capture-based sequencing (4C-seq), we showed that sex differences in distal sex-biased enhancer - promoter interactions are common. Intra-TAD loops with sex-independent cohesin-and-CTCF anchors conferred sex specificity to chromatin interactions indirectly, by insulating sex-biased enhancer - promoter contacts and by bringing sex-biased genes into closer proximity to sex-biased enhancers. Furthermore, sex-differential chromatin interactions involving sex-biased gene promoters, enhancers, and lncRNAs were associated with sex-biased binding of cohesin and/or CTCF. These studies elucidate how 3D genome organization impacts sex-biased gene expression in a non-reproductive tissue through both direct and indirect effects of cohesin and CTCF looping on distal enhancer interactions with sex-differentially expressed genes.