Immune-based anti-cancer strategies combined with radiation therapy (RT) are actively being investigated but many questions remain, such as the ideal treatment scheme and whether a potent immune response can be generated both locally and systemically. In this context, tumor-associated tertiary lymphoid structures (TLS) have become a subject of research. While TLS are present in several types of cancer with strong similarities, they are especially relevant in medullary breast carcinoma (MBC). This suggests that MBC patients are ideally suited for investigating this question and may benefit from adapted therapeutic options. As RT is a corner-stone of MBC treatment, investigating interactions between RT and TLS composition is also clinically relevant. We thus first characterized the lymphoid structures associated with MBC in a patient case report and demonstrated that they closely resemble the TLS observed in a genetical mouse model. In this model, we quantitatively and qualitatively investigated the cellular composition of the tumor-associated TLS. Finally, we investigated TLS regulation after hypo-fractionated RT and showed that RT induced their acute and transient depletion, followed by a restoration phase. This study is the first work to bring a comprehensive and timely characterization of tumor-associated TLS in basal conditions and after RT. It highlights cellular targets (i.e., Tregs) that could be selectively modulated in subsequent studies to optimize anti-tumor immune response. The study of TLS modulation is worth further investigation in the context of RT and personalized medicine.