Polygenic risk scores (PRSs) models trained from genome-wide association study (GWAS) results are set to play a pivotal role in biomedical research addressing multifactorial human diseases. The prospect of using these risk scores in clinical care and public health is generating both enthusiasm and controversy, with varying opinions among experts about their strengths and limitations. The performance of existing polygenic scores is still limited but is expected to improve with increasing GWAS sample sizes and the development of new, more powerful methods. Theoretically, the variance explained by PRS can be as high as the total additive genetic variance, but it is unclear how much of that variance has already been captured by PRS. Here, we conducted a retrospective analysis to assess progress in PRS prediction accuracy since the publication of the first large-scale GWASs, using data from six common human diseases with sufficient GWAS information. We show that although PRS accuracy has grown rapidly over the years, the pace of improvement from recent GWAS has decreased substantially, suggesting that merely increasing GWAS sample sizes may lead to only modest improvements in risk discrimination. We next investigated the factors influencing the maximum achievable prediction using whole-genome sequencing data from 125K UK Biobank participants and state-of-the-art modeling of polygenic outcomes. Our analyses suggest that increasing the variant coverage of PRS-using either more imputed variants or sequencing data-is a key component for future improvements in prediction accuracy.