Given the association of Epstein-Barr virus (EBV) with subjective perception of fatigue and demyelination in clinical conditions, the question about potential subclinical effects in the adult general population remains open. We investigate the association between individuals' EBV immune response and perceived fatigue in a community dwelling cohort (n = 864, age 62 ± 10 years old; 49% women) while monitoring brain tissue properties. Fatigue levels are assessed with the established fatigue severity scale, the EBNA-1 and VCA p18 immunoglobulin G (IgG) chronic response - with multiplex serology and the estimates of local brain volume, myelin content, and axonal density - using relaxometry- and multi-shell diffusion-based magnetic resonance imaging (MRI). In our analysis we adjust for the effects of demographic and cardiovascular risk factors, sleep apnea, depression, and polygenic risk score for multiple sclerosis. We demonstrate that EBNA-1 IgG levels are positively associated with perceived levels of fatigue, whilst VCA p18 IgG levels show a positive correlation with myelin content and a negative one with an estimate of axonal g-ratio in male participants. In the context of EBVs immune response, the polygenic risk for multiple sclerosis is not associated with increased fatigue levels, brain myelination or atrophy. Our findings bring empirical evidence about the potential role of EBVs chronic immune response in perceived fatigue and hint towards a protective role of myelination specific for men. They underscore the added value of advanced assessment of brain tissue microstructure in uncovering the mechanisms behind frequent fatigue complaints associated with EBV infection and multiple sclerosis.