Although 3D genome architecture is essential for long-range gene regulation, the significance of distal regulatory chromatin contacts is challenged by recent findings of low correlation between contact propensity and gene expression. To better understand the role of long-range interactions between distal regulatory elements during the early transformation from healthy colon to colorectal cancer, here we performed high resolution chromatin conformation capture for 33 samples including non-neoplastic mucosa, adenomatous polyps and adenocarcinomas, mostly from Familial Adenomatous Polyposis (FAP) patients. We identified hundreds of thousands of chromatin micro-structures, such as architectural stripes and loops, which originated from active cis-regulatory elements. Surprisingly, these structures progressively decayed throughout cancer progression, particularly at promoters. Meta-analyses revealed that this decay was independent of alterations in DNA methylation and chromatin accessibility. Interestingly, the degree of interaction loss was poorly correlated with gene expression changes. Instead, genes whose expression were disproportionately lower and higher than their relative promoter interaction in mucosa shifted their expression in polyps and adenocarcinomas to yield a more direct relationship between strength of interaction and gene expression. Our work provides the first high resolution 3D conformation maps during early cancer formation and progression, and provides novel insights into transcriptional readouts associated with fine-scale chromatin conformation alterations.