Using a game-theoretical approach, we investigate the dispersal patterns expected if inbreeding avoidance were the only reason for dispersal. The evolutionary outcome is always complete philopatry by one sex. The rate of dispersal by the other sex depends on patch size and mating system, as well as inbreeding and dispersal costs. If such costs are sex independent, then two stable equilibria coexist (male or female philopatry), with symmetric domains of attraction. Which sex disperses is determined entirely by history, genetic drift, and gene flow. An asymmetry in costs makes one domain of attraction extend at the expense of the other. In such a case, the dispersing sex might also be, paradoxically, the one that incurs the higher dispersal costs. As asymmetry increases, one equilibrium eventually disappears, which may result in a sudden evolutionary shift in the identity of the dispersing sex. Our results underline the necessity to control for phylogenetic relationships (e.g., through the use of independent-comparisons methods) when investigating empirical trends in dispersal. Our model also makes quantitative predictions on the rate of dispersal by the dispersing sex and suggests that inbreeding avoidance may only rarely be the sole reason for dispersal.