The family Orthopoxviridae comprises several members that can cause disease in humans, including Small-, Cow-, Camel-, and Monkeypoxvirus. In 2022 a global outbreak of the zoonotic monkeypoxvirus started with human- to- human transmission and more than 100 000 global cases in all continents. This outbreak emphasizes the need for more basic research to get a better understanding of poxvirus biology. We characterized the cellular response to poxvirus infection and performed bulk RNA-sequencing of primary human fibroblasts infected with Modified-Vaccinia Ankara Virus (MVA), Vaccinia Virus, Cowpoxvirus, Camelpoxvirus and 4 different Monkeypoxvirus (MPXV) isolates (clade 1-2b) at various time points to compare the transcriptome of infected cells. We could demonstrate that MVA infection leads to a potent activation of innate immune genes that is absent in other samples. Camelpoxvirus infection results in a strong activation of NF-kB signaling 2h post infection whereas all other viruses inhibit NF-kB activation. In addition, we observed transport of cellular proteins like histone H1, lamin A/C, emerin and topoisomerase II A/B from the nucleus into cytoplasmic poxvirus factories. This correlates with drastic structural changes inside the nucleus in electron microscopy, namely formation of nuclear filaments, indicating intensive remodeling of the nuclear architecture. Accordingly, inhibition of topoisomerase 2 using the clinically approved topoisomerase inhibitor epirubicin reduced viral replication of several MPXV strains. We observe a reduction of viral replication by epirubicin in cells of human and mouse species, indicating that the recruitment of TOP2B into viral factories is crucial for MPXV replication.