Global prevalence of Alzheimer's Disease has a strong sex bias, with women representing approximately two-thirds of the patients. Yet, the role of sex-specific risk factors during midlife, including hormone replacement therapy (HRT) and their interaction with other major risk factors for Alzheimer's Disease, such as apolipoprotein E (APOE)-e4 genotype and age, on brain health remains unclear. We investigated the relationship between HRT (i.e., use, age of initiation and duration of use) and brain health (i.e., cognition and regional brain volumes). We then consider the multiplicative effects of HRT and APOE status (i.e., e2/e2, e2/e3, e3/e3, e3/e4 and e4/e4) via a two-way interaction and subsequently age of participants via a three-way interaction. Women from the UK Biobank with no self-reported neurological conditions were included (N = 207,595 women, mean age = 56.25 years, standard deviation = 8.01 years). Generalised linear regression models were computed to quantify the cross-sectional association between HRT and brain health, while controlling for APOE status, age, time since attending centre for completing brain health measure, surgical menopause status, smoking history, body mass index, education, physical activity, alcohol use, ethnicity, socioeconomic status, vascular/heart problems and diabetes diagnosed by doctor. Analyses of structural brain regions further controlled for scanner site. All brain volumes were normalised for head size. Two-way interactions between HRT and APOE status were modelled, in addition to three-way interactions including age. Results showed that women with the e4/e4 genotype who have used HRT had 1.82% lower hippocampal, 2.4% lower parahippocampal and 1.24% lower thalamus volumes than those with the e3/e3 genotype who had never used HRT. However, this interaction was not detected for measures of cognition. No clinically meaningful three-way interaction between APOE, HRT and age was detected when interpreted relative to the scales of the cognitive measures used and normative models of ageing for brain volumes in this sample. Differences in hippocampal volume between women with the e4/e4 genotype who have used HRT and those with the e3/e3 genotype who had never used HRT are equivalent to approximately 1-2 years of hippocampal atrophy observed in typical health ageing trajectories in midlife (i.e., 0.98%-1.41% per year). Effect sizes were consistent within APOE e4/e4 group post hoc sensitivity analyses, suggesting observed effects were not solely driven by APOE status and may, in part, be attributed to HRT use. Although, the design of this study means we cannot exclude the possibility that women who have used HRT may have a predisposition for poorer brain health.