Stable social groups usually consist of families. However, recent studies have revealed higher level social structure, with interactions between family groups across different levels of social organization in multiple species. The explanations for why this apparently paradoxical behaviour arises appear to be varied and remain untested. Here, we use automated radio-tagging data from over 1000 wasps from 93 nests and social network analyses of over 30 000 nest visitation records to describe and explain interactions across levels of social organization in the eusocial paper wasp Polistes canadensis. We detected three levels of social organization (nest, aggregation and community) which exchange 'drifter' individuals within and between levels. The highest level (community) may be influenced by the patchiness of high-quality nesting habitats in which these insects exist. Networks of drifter movements were explained by the distance between nests, the group size of donor nests and the worker-to-brood ratios on donor and recipient nests. These findings provide some explanation for the multi-level social interactions, which may otherwise seem paradoxical. Fitness benefits across multiple levels of social organization should be considered when trying to understand animal societies.