Ever since its development in the 1930's, the live-attenuated Yellow Fever virus vaccine YF-17D has been highly effective. Despite the increasing knowledge on the immune biology of the YF-17D vaccine, most studies have focused only on a few types of immune cells and pathways or mainly on the primary adaptive immune response to YF-17D vaccination. Here, we examined humoral, innate and adaptive cellular responses in a longitudinal YF-17D vaccination study in Switzerland, comparing both primary and booster vaccination. In contrast to the strong innate and adaptive immune response to the primary vaccination, we find that the response to boosting is much reduced. Our data show an inverse association of neutralizing antibodies at baseline with vaccine virus replication and with the immune response upon boosting. These results suggest that booster vaccination may not have major immunological effects when neutralizing antibodies are present. Importantly, our study population was healthy adults in a non-endemic country and ultimately booster vaccine requirement must be assessed based on additional epidemiological and public health considerations in endemic areas.