Theory predicts that males adapt to sperm competition by increasing their investment in testis mass to transfer larger ejaculates. Experimental and comparative data support this prediction. Nevertheless, the relative importance of sperm competition in testis size evolution remains elusive, because experiments vary only sperm competition whereas comparative approaches confound it with other variables, in particular male mating rate. We addressed the relative importance of sperm competition and male mating rate by taking an experimental evolution approach. We subjected populations of Drosophila melanogaster to sex ratios of 1:1, 4:1, and 10:1 (female:male). Female bias decreased sperm competition but increased male mating rate and sperm depletion. After 28 generations of evolution, males from the 10:1 treatment had larger testes than males from other treatments. Thus, testis size evolved in response to mating rate and sperm depletion, not sperm competition. Furthermore, our experiment demonstrated that drift associated with sex ratio distortion limits adaptation; testis size only evolved in populations in which the effect of sex ratio bias on the effective population size had been compensated by increasing the numerical size. We discuss these results with respect to reproductive evolution, genetic drift in natural and experimental populations, and consequences of natural sex ratio distortion.