Decades of research have increased the understanding of the contribution of the anterior temporal lobes (ATLs) to semantic cognition. Nonetheless, whether semantic processing of different types of information show a selective relationship with left and right ATLs, or whether semantic processing in the ATLs is independent of the modality of the input is currently unknown. There exists evidence supporting each of these alternatives. A fundamental objection to these findings is that they were obtained from studies with patients with brain damage affecting extensive regions, sometimes bilaterally. In the current study, we assessed a group of 38 temporal lobe epilepsy (TLE) patients with either left or right small epileptogenic lesions with a battery of commonly used semantic tasks that tested verbal and non-verbal semantic processing. We found that left TLE patients exhibited worse performance than controls on the verbal semantic tasks, as expected, but also on the non-verbal semantic task. On the other hand, performance of the right TLE group did not differ from controls on the non-verbal task, but was worse on a semantic fluency task. When performance between patient groups was compared, we found that left TLE not only did worse than right TLE on the naming task, but also on the non-verbal associative memory task. When considered together, current data do not support a strong view of input modality differences between left and right ATLs. Additionally, they provide evidence indicating that the left and right ATLs do not make similar contributions to a singular functional system for semantic representation.