Sexual antagonism (SA) occurs when an allele that is beneficial to one sex, is detrimental to the other. This conflict can result in balancing, directional, or disruptive selection acting on SA alleles. A body of theory predicts the conditions under which sexually antagonistic mutants will invade and be maintained in stable polymorphism under balancing selection. There remains, however, considerable debate over the distribution of SA genetic variation across autosomes and sex chromosomes, with contradictory evidence coming from data and theory. In this article, we investigate how the interplay between selection and genetic drift will affect the genomic distribution of sexually antagonistic alleles. The effective population sizes can differ between the autosomes and the sex chromosomes due to a number of ecological factors and, consequently, the distribution of SA genetic variation in genomes. In general, we predict the interplay of SA selection and genetic drift should lead to the accumulation of SA alleles on the X in male heterogametic (XY) species and, on the autosomes in female heterogametic (ZW) species, especially when sexual competition is strong among males.