Elevated cortisol levels have been associated with poorer cognitive performance in cross-sectional studies; this may be both a factor contributing to neurodegeneration and cognitive decline and a result of developing brain pathologies. However, it is still unclear (1) whether cortisol measures predict later cognitive decline and (2) whether cortisol changes over the years might be associated with cognitive changes. We analyzed data from CoLaus/PsyCoLaus, a prospective population-based study. Salivary cortisol (4 different measures on 1 day) and neuropsychological assessments were performed at a first visit and a follow-up visit 5 years later in 625 dementia-free participants aged ≥65 years. Salivary cortisol levels at waking and 30 minutes after waking, as well as longitudinal changes in cortisol 30 minutes after waking, cortisol awakening response, and cortisol AM-PM difference were associated with decline in global cognition. After controlling for potential confounders, only longitudinal changes in cortisol 30 minutes after waking remained associated with cognitive decline. These mostly negative findings indicate absent or subtle association between salivary cortisol and cognitive decline.