Chromatin organization in the nucleus provides a vast repertoire of information in addition to that encoded genetically. Understanding how this organization impacts genome stability and influences cell fate and tumorigenesis is an area of rapid progress. Considering the nucleosome, the fundamental unit of chromatin structure, the study of histone variants (the bricks) and their selective loading by histone chaperones (the architects) is particularly informative. Here, we report recent advances in understanding how relationships between histone variants and their chaperones contribute to tumorigenesis using cell lines and Xenopus development as model systems. In addition to their role in histone deposition, we also document interactions between histone chaperones and other chromatin factors that govern higher-order structure and control DNA metabolism. We highlight how a fine-tuned assembly line of bricks (H3.3 and CENP-A) and architects (HIRA, HJURP, and DAXX) is key in adaptation to developmental and pathological changes. An example of this conceptual advance is the exquisite sensitivity displayed by p53-null tumor cells to modulation of HJURP, the histone chaperone for CENP-A (CenH3 variant). We discuss how these findings open avenues for novel therapeutic paradigms in cancer care.