The aim of this study was to examine the associations between serum lipid levels and cognitive function in a community-based sample of non-demented subjects aged 65 years and over. Participants were 2737 men and 4118 women from a population-based cohort recruited from three French cities. Visual memory, verbal fluency, psychomotor speed, and executive abilities were evaluated at baseline, and after 2, 4, and 7 years of follow-up. Lipid levels were evaluated at baseline. Multiadjusted Cox models stratified by gender were adjusted for sociodemographic and lifestyle characteristics, mental and physical health, and genetic vulnerability to dyslipidemia (apolipoprotein E and A, and cholesteryl ester transfer protein) and taking into account baseline vascular pathologies. In men, a hypercholesterolemic pattern in late-life (high total cholesterol (T-C), low HDL-C, high LDL-C levels) was associated with a 25 to 50% increased risk of decline over 7 years in psychomotor speed, executive abilities, and verbal fluency. Specific associations with low T-C and low LDL-C levels were also observed which may depend on genetic vulnerability to dyslipidemia (related to apolipoprotein A5 and cholesteryl exchange transfer protein). In contrast, in women, a 30% higher rate of decline was found in psychomotor speed with high HDL-C levels and in executive abilities with low levels of LDL-C and triglycerides, in interaction with hormonal treatment. For men and women, vascular pathologies only slightly outweighed the risk related to lipids. This suggests a complex gender-specific pattern of cognitive decline involving genetic vulnerability in men and hormonal status in women.