Despite known histological, biological, and clinical differences between lung adenocarcinoma (LUAD) and squamous cell carcinoma (LUSC), relatively little is known about the spatial differences in their corresponding immune contextures. Our study of over 1000 LUAD and LUSC tumors revealed that computationally derived patterns of tumor-infiltrating lymphocytes (TILs) on H&E images were different between LUAD (N = 421) and LUSC (N = 438), with TIL density being prognostic of overall survival in LUAD and spatial arrangement being more prognostically relevant in LUSC. In addition, the LUAD-specific TIL signature was associated with OS in an external validation set of 100 NSCLC treated with more than six different neoadjuvant chemotherapy regimens, and predictive of response to therapy in the clinical trial CA209-057 (n = 303). In LUAD, the prognostic TIL signature was primarily comprised of CD4 ⁺ T and CD8 ⁺ T cells, whereas in LUSC, the immune patterns were comprised of CD4 ⁺ T, CD8 ⁺ T, and CD20 ⁺ B cells. In both subtypes, prognostic TIL features were associated with transcriptomics-derived immune scores and biological pathways implicated in immune recognition, response, and evasion. Our results suggest the need for histologic subtype-specific TIL-based models for stratifying survival risk and predicting response to therapy. Our findings suggest that predictive models for response to therapy will need to account for the unique morphologic and molecular immune patterns as a function of histologic subtype of NSCLC.