The aim of this prospective cohort study was to evaluate the effects of lipid lowering agent (LLA) intake on cognitive function in 6,830 community-dwelling elderly persons. Cognitive performance (global cognitive functioning, visual memory, verbal fluency, psychomotor speed, and executive function), clinical diagnosis of dementia, and fibrate and statin use, were evaluated at baseline, and 2, 4, and 7 year follow-up. Multivariate Cox models were stratified by gender and adjusted for sociodemographic characteristics, mental and physical health including vascular risk factors, and genetic vulnerability (apolipoprotein E and cholesteryl ester transfer protein). For women but not men, fibrate use was specifically associated with an increased risk over 7 years of decline in visual memory only (HR = 1.29, 95% CI = 1.09-1.54, p = 0.004), and did not increase risk for incident dementia. This association was independent of genetic vulnerability related to apolipoprotein E and cholesteryl exchange transfer protein polymorphisms and occurred only in women with higher low density lipoprotein (LDL)-cholesterol levels and treated with fibrate (HR = 1.39, 95% CI = 1.08-1.79, p = 0.01) and not in those with lower LDL-cholesterol levels irrespective of fibrate treatment. For both genders, no significant associations were found between statins (irrespective of their lipophilicity) and either cognitive decline or dementia incidence. This prospective study, adjusting for multiple confounders, found no evidence that LLA given in late life reduced the risk of cognitive decline and dementia, but did raise the possibility that women with treatment-resistant high LDL-cholesterol may be at increased risk of decline in visual memory.