The many independent transitions from hermaphroditism to separate sexes (dioecy) in flowering plants and some animal clades must often have involved the emergence of a heterogametic sex-determining locus, the basis of XY and ZW sex determination (i.e., male and female heterogamety). Current estimates indicate that XY sex determination is much more frequent than ZW, but the reasons for this asymmetry are unclear. One proposition is that separate sexes evolve through the invasion of sterility mutations at closely linked loci, in which case XY sex determination evolves if the initial male sterility mutation is fully recessive. Alternatively, dioecy may evolve via the gradual divergence of male and female phenotypes, but the genetic basis of such divergence and its connection to XY and ZW systems remain poorly understood. Using mathematical modeling, we show how dioecy with XY or ZW sex determination can emerge from the joint evolution of resource allocation to male and female function with its genetic architecture. Our model reveals that whether XY or ZW sex determination evolves depends on the trade-off between allocation to male and female function, and on the mating system of the ancestral hermaphrodites, with selection for female specialization or inbreeding avoidance both favoring XY sex determination. Together, our results cast light on an important but poorly understood path from hermaphroditism to dioecy, and provide an adaptive hypothesis for the preponderance of XY systems. Beyond sex and sex determination, our model shows how ecology can influence the way selection shapes the genetic architecture of polymorphic traits.