The success of cancer immunotherapy depends in part on the strength of antigen recognition by T cells. Here, we characterize the T cell receptor (TCR) functional (antigen sensitivity) and structural (monomeric pMHC-TCR off-rates) avidities of 371 CD8 T cell clones specific for neoantigens, tumor-associated antigens (TAAs) or viral antigens isolated from tumors or blood of patients and healthy donors. T cells from tumors exhibit stronger functional and structural avidity than their blood counterparts. Relative to TAA, neoantigen-specific T cells are of higher structural avidity and, consistently, are preferentially detected in tumors. Effective tumor infiltration in mice models is associated with high structural avidity and CXCR3 expression. Based on TCR biophysicochemical properties, we derive and apply an in silico model predicting TCR structural avidity and validate the enrichment in high avidity T cells in patients' tumors. These observations indicate a direct relationship between neoantigen recognition, T cell functionality and tumor infiltration. These results delineate a rational approach to identify potent T cells for personalized cancer immunotherapy.