In plants, genes may disperse through both pollen and seeds. Here we provide a first theoretical study of the mechanisms and consequences of the joint evolution of pollen and seed dispersal. We focus on hermaphroditic self-compatible species distributed in structured populations, assuming island dispersal of pollen and seeds among small patches of plants within large populations. Three traits are studied: the rate of among-patch seed dispersal, the rate of among-patch pollen dispersal, and the rate of within-patch pollen movement. We first analytically derive the evolutionary equilibrium state of each trait, dissect the pairwise selective interactions, and describe the joint three-trait evolutionary equilibrium under the cost of dispersal and kin competition. These results are then analytically and numerically extended to the case when selfed seeds suffer from depressed competitiveness (inbreeding depression, no heterosis). Finally individual-based simulations are used to account for a more realistic model of inbreeding load. Pollen movement is shown to generate opposite selection pressures on seed dispersal depending on spatial scale: within-patch pollen movement favors seed dispersal, whereas among-patch pollen dispersal inhibits seed dispersal. Seed dispersal selects for short-distance movements of pollen and it selects against long-distance dispersal. These interactions shape the joint evolution of these traits. Kin competition favors among-patch seed dispersal over among-patch pollen dispersal for low costs of within-patch pollen movement (and vice versa for significant costs of within-patch pollen movement). Inbreeding depression favors allogamy through high rates of within- and among-patch pollen movement. Surprisingly, it may select either for or against seed dispersal depending on the cost of among-patch pollen dispersal. Heterosis favors increased among-patch dispersal through pollen and seeds. But because these two stages inhibit each other, their joint evolution might lead to decreased seed dispersal in the presence of heterosis. Of crucial importance are the costs of dispersal.