Dioecious plants are frequently sexually dimorphic. Such dimorphism, which reflects responses to selection acting in opposite directions for male and female components of fitness, is commonly thought to emerge after separate sexes evolved from hermaphroditism. But associations between allocation to male and female function and traits under sexual conflict may well also develop in hermaphroditic ancestors. Here, we show that variation in sex allocation and a trait under sexual conflict inevitably generates an advantage to sexual specialisation, fueling the transition to dioecy. In the absence of constraints, this leads to the joint evolution of separate sexes and sexual dimorphism through the build-up of an association between sex allocation and the conflict trait, such that eventually the population consists of unisexuals expressing their sex-specific optima. We then investigate how such association might materialise genetically, either via recombination suppression or via sex-dependent expression, and show that the genetic architecture of sex alloca- tion and the conflict trait readily evolves to produce the association favoured by selection. Finally and in agreement with previous theory, we demonstrate that limited dispersal and self-fertilisation, which are pervasive ecological char- acteristics of flowering plants, can offset the advantage of sexual specialisation generated by sexual conflict and thus maintain hermaphroditism. Taken together, our results indicate that advantages to sexual specialisation are inevitable when there is conflict between sexual functions in hermaphrodites, but these advantages can be counterbalanced by ecological benefits of hermaphroditism.