Cardiometabolic risk (CMR) factors are associated with accelerated brain aging and increased risk for sex-dimorphic illnesses such as Alzheimer's disease (AD). Yet, it is unknown how CMRs interact with sex and apolipoprotein E-ϵ4 (APOE4), a known genetic risk factor for AD, to influence brain age across different life stages. Using age prediction based on multi-shell diffusion-weighted imaging data in 21,308 UK Biobank participants, we investigated whether associations between white matter Brain Age Gap (BAG) and body mass index (BMI), waist-to-hip ratio (WHR), body fat percentage (BF%), and APOE4 status varied (i) between males and females, (ii) according to age at menopause in females, and (iii) across different age groups in males and females. We report sex differences in associations between BAG and all three CMRs, with stronger positive associations among males compared to females. Independent of APOE4 status, higher BAG (older brain age relative to chronological age) was associated with greater BMI, WHR, and BF% in males, whereas in females, higher BAG was associated with greater WHR, but not BMI and BF%. These divergent associations were most prominent within the oldest group of females (66-81 years), where greater BF% was linked to lower BAG. Earlier menopause transition was associated with higher BAG, but no interactions were found with CMRs. In conclusion, the findings point to sex- and age-specific associations between CMRs and brain age. Incorporating sex as a factor of interest in studies addressing CMR may promote sex-specific precision medicine, consequently improving health care for both males and females.