Personal and vicarious experience of pain activate partially overlapping brain networks. This brain activity is further modulated by low- and high-order factors, e.g., the perceived intensity of the model's pain and the model's similarity with the onlooker, respectively. We investigated which specific aspect of similarity modulates such empathic reactivity, focusing on the potential differentiation between visual similarity and psychological closeness between the onlooker and different types of models. To this aim, we recorded fMRI data in neurotypical participants who observed painful and tactile stimuli delivered to an adult human hand, a baby human hand, a puppy dog paw, and an anthropomorphic robotic hand. The interaction between type of vicarious experience (pain, touch) and nature of model (adult, baby, dog, robot) showed that the right supramarginal gyrus (rSMG) was selectively active for visual similarity (more active during vicarious pain for the adult and baby models), while the anterior cingulate cortex (ACC) was more sensitive to psychological closeness (specifically linked to vicarious pain for the baby model). These findings indicate that visual similarity and psychological closeness between onlooker and model differentially affect the activity of brain regions specifically implied in encoding interindividual sharing of sensorimotor and affective aspects of vicarious pain, respectively.