Fundus images allow for non-invasive assessment of the retinal vasculature whose features provide important information on health. Using a fully automated image processing pipeline, we extract 17 different morphological vascular phenotypes, including median vessels diameter, diameter variability, main temporal angles, vascular density, central retinal equivalents, the number of bifurcations, and tortuosity, from over 130,000 fundus images of close to 72,000 UK Biobank subjects. We perform genome-wide association studies of these phenotypes. From this, we estimate their heritabilities, ranging between 5 and 25%, and genetic cross-phenotype correlations, which mostly mirror the corresponding phenotypic correlations, but tend to be slightly larger. Projecting our genetic association signals onto genes and pathways reveals remarkably low overlap suggesting largely decoupled mechanisms modulating the different phenotypes. We find that diameter variability, especially for the veins, associates with diseases including heart attack, pulmonary embolism, and age of death. Mendelian Randomization analysis suggests a causal influence of blood pressure and body mass index on retinal vessel morphology, among other results. We validate key findings in two independent smaller cohorts. Our analyses provide evidence that large-scale analysis of image-derived vascular phenotypes has sufficient power for obtaining functional and causal insights into the processes modulating the retinal vasculature.