Many cognitive theories of semantic organization stem from reports of patients with selective, category-specific deficits for particular classes of objects (e.g., fruit). The anatomical assumptions underlying the competing claims can be evaluated with functional neuroimaging but the findings to date have been inconsistent and insignificant when standard statistical criteria are adopted. We hypothesized that category differences in functional brain responses might be small and task dependent. To test this hypothesis, we entered data from seven PET studies into a single multifactorial design which crossed category (living vs man-made) with a range of tasks. Reliable category-specific effects were observed but only for word retrieval and semantic decision tasks. Living things activated medial aspects of the anterior temporal poles bilaterally while tools activated a left posterior middle temporal region. These category-by-task interactions provide robust evidence for an anatomical double dissociation according to category and place strong constraints on cognitive theories of the semantic system. Furthermore they reconcile some of the apparent inconsistencies between lesion studies and functional neuroimaging data.