Altered reproductive hormone levels have been associated with the pathophysiology of depressive disorders and this risk may be imparted by their modulatory effect upon hippocampal structure and function. Currently it is unclear whether altered levels of reproductive hormones are causally associated with hippocampal volume reductions and the risk of depressive disorders. Here, we utilize genome-wide association study (GWAS) summary statistics from a GWAS focusing on reproductive hormones, consisting of 2913 individuals. Using this data, we generated polygenic risk scores (PRS) for estradiol, progesterone, prolactin and testosterone in the European RADIANT cohort consisting of 176 postpartum depression (PPD) cases (100% female, mean age: 41.6 years old), 2772 major depressive disorder (MDD) cases (68.6% female, mean age: 46.9 years old) and 1588 control participants (62.5% female, mean age: 42.4 years old), for which there was also a neuroimaging subset of 111 individuals (60.4% female, mean age: 50.0 years old). Only the best-fit PRS for estradiol showed a significant negative association with hippocampal volume, as well as many of its individual subfields; including the molecular layer and granule cell layer of the dentate gyrus, subiculum, CA1, CA2/3 and CA4 regions. Interestingly, several of these subfields are implicated in adult hippocampal neurogenesis. When we tested the same estradiol PRS for association with case-control status for PPD or MDD there was no significant relationship observed. Here, we provide evidence that genetic risk for higher plasma estradiol is negatively associated with hippocampal volume, but this does not translate into an increased risk of MDD or PPD. This work suggests that the relationship between reproductive hormones, the hippocampus, and depression is complex, and that there may not be a clear-cut pathway for etiology or risk moderation.