Deriving reliable information about the structural and functional architecture of the brain in vivo is critical for the clinical and basic neurosciences. In the new era of large population-based datasets, when multiple brain imaging modalities and contrasts are combined in order to reveal latent brain structural patterns and associations with genetic, demographic and clinical information, automated and stringent quality control (QC) procedures are important. Diffusion magnetic resonance imaging (dMRI) is a fertile imaging technique for probing and visualising brain tissue microstructure in vivo, and has been included in most standard imaging protocols in large-scale studies. Due to its sensitivity to subject motion and technical artefacts, automated QC procedures prior to scalar diffusion metrics estimation are required in order to minimise the influence of noise and artefacts. However, the QC procedures performed on raw diffusion data cannot guarantee an absence of distorted maps among the derived diffusion metrics. Thus, robust and efficient QC methods for diffusion scalar metrics are needed. Here, we introduce Fast qualitY conTrol meThod foR derIved diffUsion Metrics (YTTRIUM), a computationally efficient QC method utilising structural similarity to evaluate diffusion map quality and mean diffusion metrics. As an example, we applied YTTRIUM in the context of tract-based spatial statistics to assess associations between age and kurtosis imaging and white matter tract integrity maps in U.K. Biobank data (n = 18,608). To assess the influence of outliers on results obtained using machine learning (ML) approaches, we tested the effects of applying YTTRIUM on brain age prediction. We demonstrated that the proposed QC pipeline represents an efficient approach for identifying poor quality datasets and artefacts and increases the accuracy of ML based brain age prediction.