The suppression of recombination during sex-chromosome evolution is thought to be favoured by linkage between the sex-determining locus and sexually antagonistic loci, and leads to the degeneration of the chromosome restricted to the heterogametic sex. Despite substantial evidence for genetic degeneration at the sequence level, the phenotypic effects of the earliest stages of sex-chromosome evolution are poorly known. Here, we compare the morphology, viability and fertility between XY and YY individuals produced by crossing seed-producing males in the dioecious plant Mercurialis annua, which has young sex chromosomes with limited X-Y sequence divergence. We found no significant difference in viability or vegetative morphology between XY and YY males. However, electron microscopy revealed clear differences in pollen anatomy, and YY males were significantly poorer sires in competition with their XY counterparts. Our study suggests either that the X chromosome is required for full male fertility in M. annua, or that male fertility is sensitive to the dosage of relevant Y-linked genes. We discuss the possibility that the maintenance of male-fertility genes on the X chromosome might have been favoured in recent population expansions that selected for the ability of females to produce pollen in the absence of males.