The androgen receptor (AR), oestrogen receptor alpha (ESR1) and oestrogen receptor beta (ESR2) play essential roles in mediating the effect of sex hormones on sex differences in the brain. Using Voxel-based morphometry (VBM) and gene sizing in two independent samples (discovery n ​= ​173, replication ​= ​61), we determine the common and unique influences on brain sex differences in grey (GM) and white matter (WM) volume between repeat lengths (n) of microsatellite polymorphisms AR(CAG)n, ESR1(TA)n and ESR2(CA)n. In the hypothalamus, temporal lobes, anterior cingulate cortex, posterior insula and prefrontal cortex, we find increased GM volume with increasing AR(CAG)n across sexes, decreasing ESR1(TA)n across sexes and decreasing ESR2(CA)n in females. Uniquely, AR(CAG)n was positively associated with dorsolateral prefrontal and orbitofrontal GM volume and the anterior corona radiata, left superior fronto-occipital fasciculus, thalamus and internal capsule WM volume. ESR1(TA)n was negatively associated with the left superior corona radiata, left cingulum and left inferior longitudinal fasciculus WM volume uniquely. ESR2(CA)n was negatively associated with right fusiform and posterior cingulate cortex uniquely. We thus describe the neuroanatomical correlates of three microsatellite polymorphisms of steroid hormone receptors and their relationship to sex differences.