Genes are commonly found together on the same chromosome over vast evolutionary distances. This extensive physical gene linkage, known as macrosynteny, can be seen between bilaterian phyla as divergent as Chordata, Echinodermata, Mollusca, and Nemertea and likely reflects the importance of genome organization to gene regulatory landscapes. Here, we report a very different pattern of genome evolution in Bryozoa, an understudied yet ecologically important phylum of colonial invertebrates. Using comparative genomics, we reconstruct the chromosomal evolutionary history of five bryozoans. We infer the ancestral bryozoan genome organization and identify multiple ancient chromosome fusions followed by gene mixing, leading to the near-complete loss of bilaterian linkage groups. A second wave of rearrangements, including chromosome fission, occurred independently in two bryozoan classes, further shuffling bryozoan genomes. We also discover nine chromosomal fusion events shared between bryozoans and brachiopods, supporting the traditional yet highly debated Lophophorata hypothesis. Finally, we show that chromosome fusion and fission processes led to the separation of bryozoan Hox clusters. These findings demonstrate how the canonical bilaterian genome structure has been lost across an entire phylum, suggest a modification to gene regulatory landscapes, and provide a powerful source of phylogenetic information.