Models of mating-system evolution emphasize the importance of frequency-dependent interactions among mating partners. It is also known that outcross siring success and the selfing rate in self-compatible hermaphrodites can be density dependent. Here, we use array experiments to show that the mating system (i.e., the outcrossing rate) and the siring success of morphs with divergent sex allocation strategies are both density dependent and frequency dependent in androdioecious populations of the wind-pollinated, annual plant Mercurialis annua. In particular, the outcrossing rate is a decreasing function of the mean interplant distance, regulated by a negative exponential pollen fall-off curve. Our results indicate that pollen dispersed from a male inflorescence are over 60% more likely to sire outcrossed progeny than equivalent pollen dispersed from hermaphrodites, likely due to the fact that males, but not hermaphrodites, disperse their pollen from erect inflorescence stalks. Because of this difference, and because males of M. annua produce much more pollen than hermaphrodites, the presence of males in the experimental arrays reduced both the selfing rate and the outcross siring success of hermaphrodites. We use our results to infer a density threshold below which males are unable to persist with hermaphrodites but above which they can invade hermaphroditic populations. We discuss our findings in the context of a metapopulation model, in which males can only persist in well-established populations but are excluded from small, sparse populations, for example, in the early stages of colonization.