Chromosomes have an intrinsic tendency to segregate into compartments, forming long-distance contacts between loci of similar chromatin states. How genome compartmentalization is regulated remains elusive. Here, comparison of mouse ground-state embryonic stem cells (ESCs) characterized by open and active chromatin, and advanced serum ESCs with a more closed and repressed genome, reveals distinct regulation of their genome organization due to differential dependency on BAZ2A/TIP5, a component of the chromatin remodeling complex NoRC. On ESC chromatin, BAZ2A interacts with SNF2H, DNA topoisomerase 2A (TOP2A) and cohesin. BAZ2A associates with chromatin sub-domains within the active A compartment, which intersect through long-range contacts. We found that ground-state chromatin selectively requires BAZ2A to limit the invasion of active domains into repressive compartments. BAZ2A depletion increases chromatin accessibility at B compartments. Furthermore, BAZ2A regulates H3K27me3 genome occupancy in a TOP2A-dependent manner. Finally, ground-state ESCs require BAZ2A for growth, differentiation, and correct expression of developmental genes. Our results uncover the propensity of open chromatin domains to invade repressive domains, which is counteracted by chromatin remodeling to establish genome partitioning and preserve cell identity.