Seminal studies revealed differences between the effect of adaptation to left- vs. right-deviating prisms (L-PA, R-PA) in normal subjects. Whereas L-PA leads to neglect-like shift in attention, demonstrated in numerous visuo-spatial and cognitive tasks, R-PA has only minor effects in specific aspects of a few tasks. The paucity of R-PA effects in normal subjects contrasts with the striking alleviation of neglect symptoms in patients with right hemispheric lesions. Current evidence from activation studies in normal subjects highlights the contribution of regions involved in visuo-motor control during prism exposure and a reorganization of spatial representations within the ventral attentional network (VAN) after the adaptation. The latter depends on the orientation of prisms used. R-PA leads to enhancement of the ipsilateral visual and auditory space within the left inferior parietal lobule (IPL), switching thus the dominance of VAN from the right to the left hemisphere. L-PA leads to enhancement of the ipsilateral space in right IPL, emphasizing thus the right hemispheric dominance of VAN. Similar reshaping has been demonstrated in patients. We propose here a model, which offers a parsimonious explanation of the effect of L-PA and R-PA both in normal subjects and in patients with hemispheric lesions. The model posits that prismatic adaptation induces instability in the synaptic organization of the visuo-motor system, which spreads to the VAN. The effect is lateralized, depending on the side of prism deviation. Successful pointing with prisms implies reaching into the space contralateral, and not ipsilateral, to the direction of prism deviation. Thus, in the hemisphere contralateral to prism deviation, reach-related neural activity decreases, leading to instability of the synaptic organization, which induces a reshuffling of spatial representations in IPL. Although reshuffled spatial representations in IPL may be functionally relevant, they are most likely less efficient than regular representations and may thus cause partial dysfunction. The former explains, e.g., the alleviation of neglect symptoms after R-PA in patients with right hemispheric lesions, the latter the occurrence of neglect-like symptoms in normal subjects after L-PA. Thus, opting for R- vs. L-PA means choosing the side of major IPL reshuffling, which leads to its partial dysfunction in normal subjects and to recruitment of alternative or enhanced spatial representations in patients with hemispheric lesions.