Aging is accompanied by increasing difficulty in working memory associated with the temporary storage and processing of goal-relevant information. Face recognition plays a preponderant role in human behavior, and one might therefore suggest that working memory for faces is spared from age-related decline compared to socially less important visual stimulus material. To test this hypothesis, we performed working memory (n-back) tasks with two different visual stimulus types, namely faces and doors, and compared them to tasks with primarily verbal material, namely letters. Age-related reaction time slowing was comparable for all three stimulus types, supporting hypotheses on general cognitive and motor slowing. In contrast, performance substantially declined with age for faces and doors, but little for letters. Working memory for faces resulted in significantly better performance than that for doors and was more sensitive to on-line manipulation errors such as the temporal order. All together, our results show that even though face perception might play a specific role in visual processing, visual working memory for faces undergoes the same age-related decline as it does for socially less relevant visual material. Moreover, these results suggest that working memory decline cannot be solely explained by increasing vulnerability in prefrontal cortex related to executive functioning, but indicate an age-related decrease in a visual short-term buffer, possibly located in the temporal cortex.