Human altruism shaped our evolutionary history and pervades social and political life. There are, however, enormous individual differences in altruism. Some people are almost completely selfish, while others display strong altruism, and the factors behind this heterogeneity are only poorly understood. We examine the neuroanatomical basis of these differences with voxel-based morphometry and show that gray matter (GM) volume in the right temporoparietal junction (TPJ) is strongly associated with both individuals' altruism and the individual-specific conditions under which this brain region is recruited during altruistic decision making. Thus, individual differences in GM volume in TPJ not only translate into individual differences in the general propensity to behave altruistically, but they also create a link between brain structure and brain function by indicating the conditions under which individuals are likely to recruit this region when they face a conflict between altruistic and selfish acts.