Fragile X-associated tremor/ataxia syndrome (FXTAS) is a late-onset movement disorder associated with FMR1 premutation alleles. Asymptomatic premutation (aPM) carriers have preserved cognitive functions, but they present subtle executive deficits. Current efforts are focusing on the identification of specific cognitive markers that can detect aPM carriers at higher risk of developing FXTAS. This study aims at evaluating verbal memory and executive functions as early markers of disease progression while exploring associated brain structure changes using diffusion tensor imaging. We assessed 30 aPM men and 38 intrafamilial controls. The groups perform similarly in the executive domain except for decreased performance in motor planning in aPM carriers. In the memory domain, aPM carriers present a significant decrease in verbal encoding and retrieval. Retrieval is associated with microstructural changes of the white matter (WM) of the left hippocampal fimbria. Encoding is associated with changes in the WM under the right dorsolateral prefrontal cortex, a region implicated in relational memory encoding. These associations were found in the aPM group only and did not show age-related decline. This may be interpreted as a neurodevelopmental effect of the premutation, and longitudinal studies are required to better understand these mechanisms.