Mammalian SWI/SNF (mSWI/SNF) ATP-dependent chromatin remodeling complexes are multi-subunit molecular machines that play vital roles in regulating genomic architecture and are frequently disrupted in human cancer and developmental disorders. To date, the modular organization and pathways of assembly of these chromatin regulators remain unknown, presenting a major barrier to structural and functional determination. Here, we elucidate the architecture and assembly pathway across three classes of mSWI/SNF complexes—canonical BRG1/BRM-associated factor (BAF), polybromo-associated BAF (PBAF), and newly defined ncBAF complexes—and define the requirement of each subunit for complex formation and stability. Using affinity purification of endogenous complexes from mammalian and Drosophila cells coupled with cross-linking mass spectrometry (CX-MS) and mutagenesis, we uncover three distinct and evolutionarily conserved modules, their organization, and the temporal incorporation of these modules into each complete mSWI/SNF complex class. Finally, we map human disease-associated mutations within subunits and modules, defining specific topological regions that are affected upon subunit perturbation.