Postlingually deaf subjects learn the meaning of sounds after cochlear implantation by forming new associations between sounds and their sources. Implants generate coarse frequency responses, preventing place-coding fine enough to discriminate sounds with similar temporal characteristics, e.g., buck/duck. This limitation imposes a dependency on visual cues, e.g., lipreading. We hypothesized that cross-modal facilitation results from engagement of the visual cortex by purely auditory tasks. In four functional neuroimaging experiments, we show recruitment of early visual cortex (V1/V2) when cochlear implant users listen to sounds with eyes closed. Activity in visual cortex evolved in a stimulus-specific manner as a function of time from implantation reflecting experience-dependent adaptations in the postimplant phase.