CD8+ T cell recognition of peptide epitopes plays a central role in immune responses against pathogens and tumors. However, the rules that govern which peptides are truly recognized by existing T cell receptors (TCRs) remain poorly understood, precluding accurate predictions of neo-epitopes for cancer immunotherapy. Here, we capitalize on recent (neo-)epitope data to train a predictor of immunogenic epitopes (PRIME), which captures molecular properties of both antigen presentation and TCR recognition. PRIME not only improves prioritization of neo-epitopes but also correlates with T cell potency and unravels biophysical determinants of TCR recognition that we experimentally validate. Analysis of cancer genomics data reveals that recurrent mutations tend to be less frequent in patients where they are predicted to be immunogenic, providing further evidence for immunoediting in human cancer. PRIME will facilitate identification of pathogen epitopes in infectious diseases and neo-epitopes in cancer immunotherapy.