An important goal of functional neuroimaging has been to localize stimulus-specific processes in the brain. Numerous studies have revealed particular patterns of brain activity in different cortical areas in response to different object categories such as faces, body parts, places, words, letters and so forth. However, quite different patterns of activation have been given a similar interpretation in terms of category or domain specificity. Other characteristics than the response to the target category have sometimes been used to address whether a cortical brain area is functionally specialized for a given stimulus category, such as automatic processing [e.g. Joseph, J., Cerullo, M., Farley, A., Steinmetz, N., Mier, C., 2006. fMRI correlates of cortical specialization and generalization for letter processing. NeuroImage 32, 806-820] or assemblage [Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P., 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425-2430]. Here we frame the debate around the notions of category specificity as defined by Fodor [Fodor, J., 1983. The modularity of Mind. MIT Press, Cambridge, MA., Fodor, J., 2001. The mind doesn't work that way: the scope and limits of computational psychology "A Bradford book" MIT Press, Cambridge, MA] and argue that brain activation patterns consistent with category specificity remain to be demonstrated. We review possible alternatives and lay out the experimental conditions required for a conclusive demonstration of category-specific specialization in brain imaging studies.