(1) Background: A recent review concluded that there was no strong evidence for beneficial vitamin D effects on cardiovascular disease (CVD) risk, but whether individuals with vitamin D deficiency have a higher risk of CVD should be further studied. (2) Aims: We assessed the association between vitamin D levels and CVD events, CVD mortality, and overall mortality in a prospective population-based study in Lausanne, Switzerland. (3) Methods: A total of 5684 participants (53.6% women, 52.5 ± 10.7 years) were followed for a median of 14.4 years [interquartile range: 10.7-16.6]. Vitamin D blood levels were categorized as normal (≥75 nmol/L or 30 ng/mL), insufficient (50-74 nmol/L or 21-29 ng/mL), and deficient (<50 nmol/L or 20 ng/mL). (4) Results: In total, 568 cardiovascular events, 114 cardiovascular deaths, and 679 deaths occurred during follow-up. After multivariate analysis, vitamin D levels were negatively associated with CVD events: hazard ratio and (95% confidence interval) for a 10 nmol/L increase: 0.96 (0.92-0.99). However, no association was found for CVD [0.93 (0.84-1.04)] and overall mortality [0.98 (0.94-1.02)]. No associations were found between vitamin D categories and CVD events, 0.93 (0.71-1.22) and 1.14 (0.87-1.49); CVD deaths, 0.78 (0.41-1.50) and 1.10 (0.57-2.12); and overall mortality, 1.10 (0.82-1.48); and 1.17 (0.87-1.58) for insufficiency and deficiency, respectively. After excluding participants taking vitamin D supplements, similar results were obtained. (5) Conclusion: In this prospective population-based study, vitamin D levels were inversely associated with CVD events but not with CVD or overall mortality.