A first episode of depression after 65 years of age has long been associated with both severe macrovascular and small microvascular pathology. Among the three more frequent forms of depression in old age, post-stroke depression has been associated with an abrupt damage of cortical circuits involved in monoamine production and mood regulation. Late-onset depression (LOD) in the absence of stroke has been related to lacunes and white matter lesions that invade both the neocortex and subcortical nuclei. Recurrent late-life depression is thought to induce neuronal loss in the hippocampal formation and white matter lesions that affect limbic pathways. Despite an impressive number of magnetic resonance imaging (MRI) studies in this field, the presence of a causal relationship between structural changes in the human brain and LOD is still controversial. The present article provides a critical overview of the contribution of neuropathology in post-stroke, late-onset, and late-life recurrent depression. Recent autopsy findings challenge the role of stroke location in the occurrence of post-stroke depression by pointing to the deleterious effect of subcortical lacunes. Despite the lines of evidences supporting the association between MRI-assessed white matter changes and mood dysregulation, lacunes, periventricular and deep white matter demyelination are all unrelated to the occurrence of LOD. In the same line, neuropathological data show that early-onset depression is not associated with an acceleration of aging-related neurodegenerative changes in the human brain. However, they also provide data in favor of the neurotoxic theory of depression by showing that neuronal loss occurs in the hippocampus of chronically depressed patients. These three paradigms are discussed in the light of the complex relationships between psychosocial determinants and biological vulnerability in affective disorders.