Females tend to be smaller than males in woody dioecious plant species, but they tend to be larger in herbs. The smaller size of females in woody species has been attributed to higher reproductive costs, yet no satisfactory explanation has been provided for their larger size in herbs. Because herbs have higher nitrogen concentrations in their tissues than woody plants, and because pollen is particularly rich in nitrogen, we predicted that male growth would be more compromised by reproduction than female growth. To test this hypothesis, we conducted three experiments on the annual dioecious herb Mercurialis annua. First, we compared the timing of reproduction between males and females and found that males started flowering earlier than females; early flowering is expected to compromise growth more than later flowering. Second, we compared plants allowed to flower with those prevented from flowering by experimental debudding and found that males incurred a higher reproductive cost than females in terms of both biomass and, particularly, nitrogen. Third, we grew plants under varying levels of nitrogen availability and found that although sexual size dimorphism was unaffected by nitrogen, females, but not males, decreased their relative allocation to both roots and reproduction under high nitrogen availability. We propose that males deal with the high cost of pollen production in terms of nitrogen by allocating biomass to nitrogen-harvesting roots, whereas females pay for carbon-rich seeds and fruits by investing in photosynthetic organs. Sexual dimorphism would thus seem to be the outcome of allocation to above- versus below-ground sinks that supply resources (carbon versus nitrogen) limiting the female and male reproduction differentially.